Cell‐MICS: Detecting Immune Cells With Label‐Free Two‐Photon Autofluorescence and Deep Learning

ABSTRACT

Multiphoton imaging has been widely used for deep‐tissue imaging. Although its label‐free, metabolic contrast is ideal for investigating inflammation, the label‐free two‐photon induced autofluorescence is often regarded as less specific compared to conventional antibody markers. In this work, we investigate the potential for multiphoton imaging with computational specificity (MICS) by training a convolutional neural network on images of different immune cells. A low‐complexity squeezeNet architecture was able to achieve reliable immune cell classification results (0.89 ROC‐AUC, 0.95 PR‐AUC for binary classification between T cells and neutrophils; 0.689 F1 score, 0.697 precision, 0.748 recall for multi‐class classification between six isolated cell types). Perturbation tests confirmed that the model was not confused by the extracellular environment and that 2P‐AF from NADH and FAD is equally important for the classification. In the future, deep learning could provide computational specificity for specific immune cells in unstained tissues, with great potential for label‐free in vivo endomicroscopy.

We demonstrate multiphoton imaging with computational specificity for label‐free cell identification (cell‐MICS). Based on the intrinsic contrast of two‐photon induced autofluorescence from metabolic proteins, a deep neural network is able to distinguish various different immune cell types. The system's performance is stable and efficient. MICS generalizes extremely well even from very small data sets and could be translated for in vivo and marker‐free immune cell detection in the future.

1. Introduction

Inflammation is a key response of biological organisms to harmful stimuli, like pathogens or tissue damage. This mechanism is layered across different scales, ranging from macroscopic swelling of the 3D tissue down to the release of sub‐cellular biochemical molecules. At the microscopic level, inflammation is governed by the recruitment and activation of a range of different immune cells. This process is highly dynamic and involves a specific timing of different immune cell types that are present at different locations, at different times, and in different amounts. The initial response during acute inflammation caused by tissue damage or by an unknown pathogen is typically governed by a series of innate immune cells, while inflammation through a specific pathogen might be identified and targeted by a chain of adaptive immune cells. Anomalous adaptive immune responses can lead to chronic inflammation and are often the root of many autoimmune diseases.

Thus, the identification of immune cells is essential to monitor inflammation dynamics, enable a deeper understanding of the spatial organization of immune reactions or to diagnose autoimmune diseases and track the effect of potential therapeutics.

Ideally, such detection tools should provide microscopic, cellular resolution and minimize alterations to the target cells, that is, it is desirable to avoid fixation and sectioning as well as biochemical binding with antibody markers. Optical microscopy has developed several label‐free imaging technologies that provide contrast to the natural interaction between biochemical structures and light. Many of those techniques have already been used for detection of specific cell types, often by leveraging the predictive power of deep learning (DL) models to boost specificity.

The most accessible and straightforward label‐free technique is probably bright‐field imaging (BF), which was used as input to DL models to classify type and state of certain cells with a performance comparable to fluorescence‐based approaches [1, 2, 3]. Phase contrast microscopy is another commonly used imaging technique where DL models have shown accurate results, for instance for the segmentation of bovine aortic endothelial cells [4], classification of myoblast cells [5] or the classification of cancer cells [6]. Differential interference contrast (DIC) imaging provides enhanced contrast at edges and structural features by exploiting optical path length differences, producing pseudo‐3D relief images of unstained cells [7]. DL has enabled classification and segmentation of cells [8], their health status [9], or bacteria [10] from DIC images. Ogawa et al. compared the effect of different imaging modalities on DL‐based classification between lymphoid‐primed multipotential progenitor (LMPP) and pro‐B cells [11]. They found no significant difference between BF, phase contrast, and DIC at generally good classification performance (area under the receiver operating characteristic curve—AU‐ROC of 0.9) [11]. Quantitative phase imaging (QPI) is a more advanced computational imaging technique that can provide intrinsic quantification of the optical path length difference to provide a quantitative imaging signal with decent cellular specificity [12, 13]. QPI has been used in a wide range of applications of machine learning‐assisted cell assessment [14, 15], like classification of red blood cell morphology [16], scoring [17], or classification of cancer cells [18], distinction between healthy B cells and lymphoblasts [19], as well as classification of stages in B cell acute lymphoblastic leukemia [19]. Similar advances have also been made with label‐free digital holographic microscopy [20, 21, 22].

Despite their success in live cell microscopy, the translation of BF, phase imaging, DIC or QPI towards live tissue, 3D endo‐microscopy is less straightforward, as these techniques either show limited 3D capabilities (BF and phase) or require multiple acquisitions under controlled illumination patterns (DIC and QPI), which is more challenging in a tightly‐packed endoscope design, as well as computational reconstructions (DIC and QPI), which can be more error‐prone for noisy in vivo data. Furthermore, it is often desirable to obtain functional or metabolic information, especially for immune reactions. Although quantitative measurements of optical path length in QPI can be related to dry cell mass [23], these image quantities are only indirectly related to metabolic activities.

In contrast, multiphoton microscopy (MPM) exploits the confocal nature of nonlinear excitation for optical sectioning as well as reduced scattering at infra‐red wavelength for greater penetration depths [24]. Moreover, the label‐free measurement of natural, 2‐photon induced autofluorescence (2P‐AF) from metabolic coenzymes, like nicotinamide adenine dinucleotide (NADH, H for hydrogen), and flavin adenine dinucleotide (FAD), can directly be linked to metabolic processes and mitochondrial activity in cells [25]. These quantitative measurements have already been used to reveal cellular metabolic states [26] or mitochondrial dysfunction [27], to distinguish breast cancer cells from normal controls [25], to identify different breast cancer tissues [26], to detect Alzheimer's disease in fresh murine brain samples [27] or to support the distinction between brown and white adipose tissue [28].

This unique combination of deep tissue 3D imaging capability paired with label‐free metabolic contrast makes MPM ideal for investigating inflammation and remodeling of inflammatory tissue directly within the native 3D tissue structure [29, 30, 31, 32].

Although single‐photon induced AF was already used for classification of immune cell types [33, 34], cell sorting via flow cytometry [35, 36] or digital staining of tissue sections and live cells [37, 38, 39], 2P‐AF has only rarely been explored for the same purposes. For instance, Gehlsen et al. [40], as well as our own group [35] already suggested 2P‐AF as a potential label‐free biomarker for differentiating certain immune cells or their activation states. However, the full potential of metabolic, cellular 2P‐AF for computational specificity to immune cells, as proposed for many other imaging modalities [37, 38, 39], is still underexplored. Label‐free multimodal imaging of coherent‐anti‐stokes Raman scattering (CARS) and MPM was used for digital H&E staining based on the label‐free input [41, 42]. However, these procedures were limited to formalin‐fixed and paraffin‐embedded tissue sections and did not offer specificity to different immune cell types beyond that of H&E. The development of state‐of‐the‐art DL models for specific immune cell classification based on label‐free multiphoton images has not yet been demonstrated.

Such computational specificity for immune cells in label‐free multiphoton imaging might be particularly promising, since multiphoton imaging is already being used for in vivo imaging and endo‐microscopy (MPEM) [29, 43, 44, 45, 46, 47, 48], which enables optical histology imaging in live animal models.

In this work, we demonstrate the multiphoton imaging with computational specificity for immune cell classification (cell‐MICS, named in tribute to “phase imaging with computational specificity—PICS” [23]), which represents the very first use of DL models for the classification of label‐free 2P‐AF images of various different immune cell types. For that purpose, we used an existing data set from various different immune cell types [35] to train CNN models on automated, specific identification of immune cells. Ground truth annotations were elegantly obtained through simultaneously recorded fluorescence antibody markers, or through clear experimental designs, which circumvented the common bottleneck of manual expert annotations. Systematic perturbation tests validated robust classification performance and data‐efficient learning, while also revealing that models that focus on molecular 2P‐AF significantly outperformed those that only use spatial information on cell size and shape alone.

2. Materials and Methods

2.1. Data Set

These original images were obtained in a previous work by Lemire et al. [35] from immune cells that were isolated from the spleen (CD4⁺/CD8⁺ T cells, B cells) or bone marrow (macrophages, dendritic cells, neutrophils) of wildtype C57BL/6 mice, seeded on glass slides and imaged with a multiphoton microscope (TriMScope II, LaVision BioTec, Bielefeld, Germany) using filters that target AF from NADH (BP 450/70) and FAD (BP 560/40). The respective spectra and filter bands are shown in Figure 1a, and all acquisition settings are displayed in Table 1. In addition to these AF channels, the forward scattered Dodt channel was recorded (displayed in gray in all figures). Similar to DIC, Dodt contrast is a gradient‐based technique, and it provides optical sectioning and improved visualization of thick tissue slices [49]. Although it offers enhanced structural detail, its application for DL‐based immune cell classification is less established, and, as with DIC, it primarily encodes morphological rather than biochemical or metabolic differences. Each full raw image had a size of 1024 × 1024 pixels across a field of view (FOV) of 405 × 405 μm², containing dozens to hundreds of cells. Details of all data in this study are shown in Table 2.

FIGURE 1.

Automated immune cell identification based on label‐free 2‐Photon autofluorescence and deep learning. A scanning multiphoton microscope is used to generate label‐free image data from immune cells on a substrate. An 810 nm, ultra‐short‐pulsed laser is used to excite autofluorescence from NADH and FAD, while gradient Dodt contrast images are collected in transmission mode. Label‐free images from various immune cells are collected and used as input to a convolutional neural network, which has been trained to predict immune cell type. (a) Optical setup: laser‐scanning two‐photon microscopy, (b) dataset: label free autofluorescence and Dodt contrast, and (c) deep learning: predicting immune cell type.

TABLE 1.

Data acquisition settings.

Excitation wavelength (nm)	810
NADH emission window (nm)	415–485(BP 450/70)
FAD emission window (nm)	540–580 (BP 560/40)
Pixel size (nm)	395
Numerical aperture	0.95

TABLE 2.

Data sets. The full FOV images were cropped into patches of 64 × 64 pixels, centered around each cell.

	Binary classification	Multi‐class classification
Full FOV images	31	85
Patches per class (64 × 64 pixels)	CD4+ T cells: 1311	CD4+ T cells: 144
		C84+ T cells: 612
		B cells: 1953
	Neutrophils: 3764	Neutrophils: 366
		Dendritic cells: 140
		Macrophages: 209
Total patches (64 × 64)	5075	3424

2.1.1. Cell Mixture

First, we investigated the potential to differentiate two different cell types (neutrophils and T cells) that were present in the same sample (Figure 2). In that case, T cells were stained with the allophycocyanin (APC)‐labeled lymphocyte marker $\alpha$‐CD3 to obtain ground truth annotations. This APC signal had no significant spectral overlap with natural AF emissions (BP 675/67 for APC, see Figure 1a) and was subsequently recorded at a different excitation wavelength (810 nm for AF and 1040 nm for APC) which prevented channel leakage entirely.

FIGURE 2.

Classification results from unstained neutrophils and stained T cells in mixture. T cells were isolated, stained with an APC‐labeled $\alpha$‐CD3 marker and mixed with unstained neutrophils, before imaging. The two label‐free channels of NADH and FAD were used as input to a deep learning model, while the fluorescence channel of the specific marker was used to derive ground truth annotations for the training. Receiver operating characteristic (ROC) curve, precision‐recall (PR) curve, and confusion matrix for the validation examples of a 5‐fold CV loop indicate that the model is able to differentiate both cell types reasonably well, when using label‐free MPM images as input. All values denote the average of the validation set across 5‐folds (stratifiedgroupkfold CV), resulting in decimal fractions. Scale bar of 10 μm applies to all images. The total data set for training and cross‐validation included 3424 image patches of 64 × 64pixels. (a) Experimental protocol and (b) classification performance.

In total, this data set consisted of 31 full field‐of‐view image pairs of label‐free AF images. Four images only contained T cells, seven images only contained neutrophils, and 20 images were from samples that contained roughly a 50:50 mixture of both cell types.

An image processing procedure was developed in the open‐source image processing software Fiji to crop single cell image patches from these full‐FOV images. The image processing macro loaded the raw data and registered AF and APC images via scale invariant feature transform (SIFT) [50]. Cell detection was performed via semi‐automated, user‐validated Otsu‐thresholding of the NADH channel (“setAutoThreshold(’Otsu no‐reset’)”), auto adjusted to include 10% of bright pixels, and a human observer verified or adjusted the threshold manually, if needed. Thresholding was followed by Watershed and “Analyze Particles” (minimal size of 25 pixels area and 0.3–1 circularity). The center of the detected regions of interest (ROI) was then used to crop a patch of 64 × 64 pixels (25.6 × 25.6 μm² FOV) around it. For each image patch, the two AF channels and the Dodt channel were saved together as multi‐channel TIF file. Cells at the edges of the original image were ignored to ensure a consistent size of 64 × 64 for all patches. The respective APC channel of the patch was used for thresholding to determine it as APC‐positive or APC‐negative. This procedure resulted in a data set with a total of 5078 annotated image patches, each with a unique cell in the center.

2.1.2. Multi‐Class Data Set

In the second case, we investigated the potential of two‐photon induced AF for multi‐class classification of several different cell types. For that purpose, we used a different experimental design in the available data base [35], where cell types were not mixed, and each isolated cell type was imaged separately without antibody reference (Figure 3). Therefore, annotations for each cell type were available from the experimental protocol instead of a fluorescence antibody (see Figure 3). Purity of these isolated cell suspensions reached values of > 95% in each case [35]. In total, we pooled 85 unique full FOV images from six different cell types. These images were processed into 64 × 64 pixel patches following the same procedure as explained above, resulting in a total of 3424 cell patches.

FIGURE 3.

Classification results from six different immune cells. Each cell type was isolated and imaged separately, resulting in a separate data set for each cell type, so that ground truth annotations were available through that experimental design. Again, a deep CNN model was trained with label‐free AF images as input to predict cell type. Multi‐class classification results are evaluated by the 5‐fold cross‐validation confusion matrix and performance metrics. All values denote the average of the validation set across 5 folds (kfold CV), resulting in decimal fractions. Scale bar of 10 μm applies to all images. The total data set included 5075 image patches of 64 × 64 pixels. (a) Experimental protocol and (b) classification performance.

2.2. Model Architecture

We selected the SqueezeNet architecture as the backbone model for this study, as it is known to preserve competitive accuracy while reducing the number of trainable parameters. In the case of our relatively small data set (see Section 2.1), this might minimize the risk of overfitting [51]. We used a pretrained SqueezeNet architecture (squeezenet1.0, torchvision, pretrained on the ImageNet‐1K data set) and adjusted input, features and classifier layers (see Block 0, 1 and 11 in Table S1) to match the shape of our input data and prediction labels. SqueezeNet is comprised mainly of “Fire” modules which are essentially squeezed convolution layers that only have 1 × 1 filters, feeding into an expand layer that has a mix of 1 × 1 and 3 × 3 convolution filters (for more details, see section 5.3 in Ref. [52]). This pretrained model was then fine‐tuned with our data set for the two respective classification tasks.

2.3. Training

The entire framework for training was developed in Python 3.9 using PyTorch 2.5.1 and CUDA 11.8. A custom data loader was used to load the TIF files for each cell patch as pytorch tensors, apply data augmentation (horizontal flip, vertical flip and rotation), apply a Gaussian filter ($\sigma$ = 2) and finally, to carry out a z‐score standardization to the mean and standard deviation of the entire data set. A 5‐fold cross validation (CV) was used, where 80% of the data were used for training and the remaining 20% for validation in each fold (sklearn, model selection, Kfold). In case of the multi‐class data, a grouped stratification (sklearn, model selection, stratifiedgroupkfold) was used since this data set was severely imbalanced.

The models were trained for 300 epochs using a cross entropy loss, an Adam optimizer [53], a learning rate scheduler and stochastic weight averaging (SWA) [54]. As shown in Figure S1, this resulted in a loss convergence. These main hyperparameters are summarized in Table S2. In each fold, loss, true positives, true negatives, false positives and false negatives were tracked to calculate per‐fold performance metrics which were averaged to the final evaluation.

2.4. Data Perturbation Experiments

In order to evaluate if the model learns the desired cellular information instead of overfitting to noise or extra‐cellular background, we introduced a series of spatial perturbations in the binary classification data (see “Cell mixture” above). Similarly to the approach by Cook et al. [55], all of these perturbations were in place for the entire network training process, resulting in an independently trained model for each perturbation test. As a spatial perturbation, we defined concentric circles with fixed diameters that masked either the inside or the surrounding area outside of that circle. The circles were always centered within the image and had diameters of 5, 20, 40, and 60 pixels (see Figure 2a).

2.5. Model Perturbation Experiments

Our selected SqueezeNet architecture had a total of 735 937 trainable parameters. To evaluate whether this capacity was adequate for our tasks, we performed a model perturbation test, where certain blocks of the architecture were frozen to the initially pretrained condition, without re‐optimization through backpropagation on the new data (requires_grad = false). The first input layer and the classifier block (blocks 0 and 11 in Table S1) were always allowed for backpropagation, resulting in a minimum of 15 760 trainable parameters. We then increased the trainable capacity by subsequently “un‐freezing” one layer at a time, starting at #3 in Table S1 (28 176 parameters) until #10 (735 937 parameters).

2.6. Channel Perturbation Experiments

Finally, we evaluated the relative importance of the different input channels (representing different types of molecular/metabolic information) by training the same model architecture on four different input channel configurations: (i) NADH autofluorescence only, (ii) FAD autofluorescence only, (iii) Dodt gradient contrast only, and (iv) NADH and FAD autofluorescence. Again, an entirely new model was trained on each of these configurations. These perturbations were carried out in the binary classification data (see “Cell mixture” above).

2.7. Testing the Impact of Class Imbalance

Since the data set in the multi‐class problem is particularly imbalanced (see, Figure S1), additional tests with 10 randomly selected sub‐sets of balanced label distributions were tested. Therefore, the two largest classes of CD8⁺ T cells and B cells were downsampled to match the third‐largest class. In each of these 10 downsampled sub‐sets, a complete 5‐fold Stratified‐GroupKFold CV was performed and the mean performance across all 5 folds was used to then determine the mean and the 95% CI across the 10 sets.

2.8. Deep Contrastive Embedding of Two‐Photon Autofluorescence for Different Immune Cells

To examine class separability of two‐photon autofluorescence (2P‐AF) immune cell signatures and to identify potential overlaps in the high‐dimensional CNN feature space, uniform manifold approximation and projection (UMAP) [56] was performed on the embeddings obtained from the validation set of each trained model in the 5‐fold cross‐validation. The embeddings were extracted from the final layer preceding the classification layer, as feature spaces are fold‐specific and not mutually aligned. The first two UMAP dimensions were visualized, and data points were color‐coded according to their ground‐truth class labels for qualitative assessment.

2.9. Computational Hardware

All models were trained on a workstation equipped with NVMe SSD, Nvidia RTX 3090 GPU, and Intel Core‐i9 10850k CPU (10 cores of 3.6 GHz). Training took about 25 min (1500 s) for each of the perturbation experiments.

3. Results

Table 3 shows the main performance metrics from both tasks. The per‐class performance is shown in Table S3.

TABLE 3.

Performance metrics displayed as mean value within a 95% confidence interval (CI) from a 5‐fold cross validation.

	Binary classification	Multi‐class classification
Accuracy	0.862 (0.855–0.868)	0.799 (0.548–0.949)
ROC‐AUC	0.894 (0.883–0.904)
PR‐AUC	0.950 (0.942–0.957)
Precision	0.899 (0.884–0.915)	0.697 (0.506–0.877)
Recall	0.917 (0.903–0.935)	0.748 (0.573–0.917)
MCC	0.634 (0.623–0.647)	0.683 (0.392–0.924)
F1 (macro)	0.816 (0.812–0.823)	0.689 (0.510–0.855)

3.1. Classification of Immune Cells in Mixed Samples

Figure 2 shows the main classification results of the trained network. For the binary classification between T cells and neutrophils in mixed samples, we can report a 5‐fold average AUC‐ROC of 0.87, an AUC of the precision recall curve of 0.95, and a validation accuracy of 84.89%.

3.2. Multi‐Class Classification of Other Types of Immune Cells

The multi‐class cell classification shown in Figure 3 generally shows successful classification results. The model achieved an F1 score of 0.689 (ranges from 0 to 1), a precision of 0.697 (range from 0 to 1), a recall of 0.748 (range from 0 to 1) and a Matthew's correlation coefficient (MCC) of 0.683 (range from −1 to 1). The multi‐class accuracy was 52.67% (random guess would be 16.6%), and most of the mis‐classified examples were B cells that were falsely predicted as CD8⁺ T cells. Potential reasons for the mis‐classification could be that B cells are CD8⁺ T cells are quite similar in shape in size. Additionally, this multi‐class data set was very skewed (as seen in Figures 3 and S1), with B cells being by far the largest class, which amplifies this issue statistically. Therefore, the metrics of F1 score, precision, recall and MCC are more conclusive metrics than the accuracy in this case.

The limited size of the data set and the greater number of labels lead to the challenge of splitting examples across the five validation folds. The same K‐fold CV strategy that was used for the binary classification (see above) would have resulted in folds that lacked representation of each class, leading to per‐fold performance metrics that are likely underestimating a potential performance on larger data. This is a known challenge when splitting imbalanced data sets of multiple classes and few examples. There, we employed a grouped stratification strategy for this multi‐class problem. Although this procedure reduced the reported validation performance, we prioritized methodological rigor by enforcing strict group‐aware splitting to prevent data leakage—a conservative choice that sacrifices short‐term metric optimization for long‐term generalizability in this class‐imbalanced setting.

3.3. Perturbation Experiments

3.3.1. Data Perturbation

When presented with data that contain only a small fraction of central pixels (second bar in Figure 4a), the model performance drops from 0.87 AUC‐ROC and a 0.95 PR‐AUC (Figure 4 and the first bar in Figure 4a) to only 0.75 AUC‐ROC/0.89 PR‐AUC. When trained on data with continuously larger circular areas from the center, the performance continuously increases, as expected, approaching a performance similar to that in the unperturbed case. On the other hand, when trained on data in which central pixels were continuously removed, the performance drops sharply to only 0.67 AUC‐ROC/0.83 AUC‐PR and slowly increases when allowing for more pixels. Together, these results indicate that learning heavily relies on the actual cellular information in the center of the image patch, as intended, and is not severely influenced by background pixels which contain noise and might occasionally include other cells at the edges.

FIGURE 4.

Perturbation experiments indicate that the model is learning the desired cellular autofluorescence pattern. (a) Circles of different diameters were used to mask center or edge pixels. Examples of perturbation data are shown below for both cell types. (b) Performance changes with varying numbers of trainable parameters in the model (see Table S1). (c) Performance changes for different molecular imaging channels. All performance values show area under the curve (AUC) for receiver operating characteristic (ROC) on the left and for precision‐recall (PR) curve on the right from the validation case in a 5‐fold cross‐validation scheme. Bar height represents the average validation performance of all folds, with the standard deviation as error bar. All perturbations were carried out on the binary classification data (see “Cell mixture” above).

3.3.2. Model Perturbation

The results of our model perturbation tests, shown in Figure 4b, indicate that both ROC‐AUC and PR‐AUC steadily increase when increasing the number of trainable parameters. Moreover, this steady increase flattens out and converges at higher capacities of trainable parameters (i.e., between 571 072 and 735 937 trainable parameters), which indicates that the chosen model has an adequate capacity for the given task, and that a larger model is not expected to perform significantly better.

3.3.3. Cell Detection Is Driven by Cellular 2P‐AF, Not by Shape

Finally, we present the results of training the model on different image channels in Figure 4c. It can be observed that the performance of a model trained on only one single AF channel (i.e., NADH or FAD, respectively) is already close to the overall performance including all channels. A model that was solely trained on the Dodt signal is close to a random guess (0.5 AUC‐ROC).

3.4. Biology Rather Than a Statistical Imbalance Is the Main Driver of the Multi‐Class Prediction

Although the downsampled data set substantially reduced the total sample size by 53% from 3.424 to only 1.591, the overall results generally confirm the above results on the original data set, as seen in Table 4. The 5‐fold Stratified‐GroupKFold strategy from 10 different down‐sampled sets resulted only in a minor drop in performance, but not in a complete failure of the prediction. This minor drop in performance was expected, since (a) the data size was greatly reduced and (b) the Stratified‐GroupKFold of this smaller data set resulted in a greater number of folds with missing class labels. Consequently, the confidence intervals may appear wider, as their estimation is based on a reduced number of fully‐representative folds. This apparent inflation reflects the higher statistical uncertainty arising from missing class representations, rather than variability in model performance itself.

TABLE 4.

Performance metrics of the multi‐class task displayed as mean value within a 95% confidence interval (CI) for the original, imbalanced data set (left), and across 10 different sub‐sets that were downsampled by 53% to contain balanced label distributions.

Metric	Original imbalanced data set mean and CI from 5‐fold Stratified‐GroupKFold CV	Down‐sampled balanced data sets mean and CI from 10 randomly sampled sets each as the mean value from a 5‐fold Stratified‐GroupKFold
Accuracy	0.799 (0.548–0.949)	0.736 (0.581–0.812)
Precision	0.697 (0.506–0.877)	0.630 (0.482–0.703)
Recall	0.748 (0.573–0.917)	0.676 (0.582–0.714)
MCC	0.683 (0.392–0.924)	0.684 (0.509–0.756)
F1 (macro)	0.689 (0.510–0.855)	0.617 (0.484–0.683)

Note: In the latter case, the performance of each set was measured as the mean of the same Stratified‐GroupKFold strategy (right).

3.5. Deep Contrastive Embedding Confirms Separability of Two‐Photon Autofluorescence Signatures for Different Immune Cells

Figure 5 shows a visualization of the first two UMAP dimensions for both experiments (binary classification from cell mixture and multi‐class classification from isolated cells) for two exemplary folds. These learned deep embeddings of the CNN revealed distinct distributions for different immune cell types, with clear clustering by class and only minor overlaps. This indicates that the two‐photon autofluorescence (2P‐AF) signatures are indeed separable in the high‐dimensional feature space of the trained CNN.

FIGURE 5.

Deep contrastive embeddings confirm separability of two‐photon autofluorescence immune cell signatures across both experiments. UMAP projections of the final CNN embeddings from two representative folds (a) of the binary classification from cell mixture and (b) of the multi‐class classification from isolated cells (please note that not all folds included representation of all classes due to stratified grouped split). Each subplot shows validation embeddings from the final preclassification layer, reduced to the first two UMAP dimensions. Data points are color‐coded according to the ground‐truth immune cell class. Distinct clustering of classes within each fold demonstrates that the learned contrastive embeddings effectively capture separable 2P‐AF signatures.

4. Discussion

This study was designed to develop deep neural networks for automated identification of specific immune cell types, based on label‐free 2‐photon induced AF image data. In the binary classification of mixed T‐cells and neutrophils, we achieved a performance of 0.87 AUC‐ROC and 0.95 AUC‐PR which is on par with a recent state‐of‐the‐art method for single‐photon induced AF (0.92–1 AUC‐ROC [33]), despite the fact that we only used three input channels instead of 56 [33] and relied on a tiny data set of only 5075 single immune cells. The second case of classifying six different major immune cell types also performed reasonably well (0.689 F1 score, 0.697 precision, 0.748 recall, and 0.683 MCC), especially since this task was a more complicated multi‐class problem and since even fewer and yet more imbalanced data were available. However, we acknowledge that larger data sets would probably result in higher classification performance.

In addition to demonstrating an overall successful use of label‐free MPM and DL for immune cell identification, our systematic perturbation tests enabled further investigation of robustness of the classification, as well as of the relative importance of different spatial‐molecular patterns. The combination of controlled experiments and architectural choices ensured the model learned generalizable patterns rather than overfitting to artifacts or background noise in the limited dataset. Moreover, molecular contrast perturbations indicated that learning was significantly more successful if the model had access to the full spatial distribution of molecular 2P‐AF instead of only the spatial information from the gradient Dodt channel. While the Dodt signal of simple forward‐scattering was insufficient to differentiate T cells from similarly‐sized neutrophils (see Figure 4c), the 2P‐AF signal resulted in a greatly improved classification performance. The difference between NADH and FAD was insignificant, indicating that both channels are equally important. Additionally, it was possible to rule out statistical class imbalances as a potential driver for this prediction, and confirm that optical measurements of the underlying cellular metabolism were the cause of successful immune cell prediction by our cell‐MICS models. This highlights that the 2P‐AF from metabolic coenzymes NADH and FAD drives the DL‐based classification, which is a very plausible biological link since this metabolic activity plays a central role in immune cells [35, 40]. In the future, additional spectral resolution of these 2P‐AF signals might further improve classification performance [33].

The use of elegant data collection strategies allowed us to obtain high‐quality data labels without human annotation and thus, to bypass manual annotations which can often be the labor‐intensive bottle‐neck in developing, training and validating new DL models. The use of reference fluorescence markers in combination with AF poses the unique challenge to avoid spectral overlap to preserve the specificity of the imaging signals and prevent information leakage. The presented approach to use subsequent recording of the APC channel at a different excitation can be used for two‐photon excitation without overlapping with the natural AF emission. In the future, the use of genetically encoded reporter fluorophores would be a promising addition to provide cell annotations for the development of multiphoton imaging with computational specificity to other cells, like bacteria (e.g., Citrobacter rodentium [31]) or a finer resolution to different subtypes of immune cells (e.g., Ccr2⁺/RFP, Cd68⁺/GFP, and Cx3cr1 macrophages [32]). The use of such genetically encoded fluorophores for digital staining has already been shown in the example of digital staining of mitochondria in living cells using correlative imaging [57] and would be most suitable for in vivo imaging of immune cells in 3D tissues.

Following the good scientific practices suggested in Ref. [39], we aim to discuss the overall uncertainty of our approach, which involves the fundamental labeling uncertainty in obtaining the ground truth data annotations and the prediction uncertainty of the model. The latter can be gauged by the standard deviation in performance across the different folds which is around 1%–3% (0.025 ROC‐AUC or 0.013 PR‐AUC). The label uncertainty however, is more difficult to assess. In the multi‐class classification experiment, data labels were obtained through the experimental protocol of isolating the respective cells. It has been stated that the purity of these isolated cell samples reached > 95% [35] which can be regarded as the upper boundary for the labeling specificity for this experiment. In the case of binary classification of T cells and neutrophils, the labeling specificity is related to the biochemical binding specificity of the antibody which is much more difficult to assess quantitatively. It is widely accepted that T cells carry CD3 antigens that bind with anti‐CD3 antibodies, while neutrophils, B cells and macrophages do not. However, these statements are usually more qualitative, like “B cells, granulocytic series, and monocytes/macrophages are all CD3‐negative.” [58] or “the vast majority of mature T cells bear TCR $\mathit{\alpha \beta }$” [58]. Quantitative values for the half‐maximal effective concentration (EC50) values of certain CD3 antibodies are available in literature to range from 3 to 24 nM [59] or 17 to 161 nM [59]. However, the actual specificity in percentage (%), that is, the percentage of selective binding to T cells against unintended binding to neutrophils, was not determined in this experimental study. Nevertheless, uncertainties in the data labeling is also an extremely challenging problem to conventional, manual annotations which are often subject to human errors, individual biases, or inter‐observer variability [60].

5. Conclusion and Outlook

In summary, we demonstrated the successful use of label‐free MPM and DL for immune cell identification by training a CNN on label‐free 2P‐AF image data while obtaining reliable ground truth labels from the experimental design, without manual annotations. Our simple yet effective model achieved good performance for differentiating T cells and neutrophils (0.87 AUC‐ROC and 0.95 AUC‐PR), as well as for multi‐class classification of CD4⁺ T cells, CD8⁺ T cells, B cells, neutrophils, dendritic cells and macrophages (0.689 F1 score, 0.697 precision, 0.748 recall, 0.683 MCC). A series of different perturbation experiments shows that the model is stable (i.e., not confused by extracellular environment), efficient (i.e., has an adequate number of trainable parameters and learns from very small data sets), and that the detection is driven by cellular 2P‐AF (NADH and FAD) instead of gradient Dodt contrast.

Computational specificity to immune cells, as proposed here, would be a promising addition to label‐free multiphoton imaging. In the future, this concept could be applied to 3D deep‐tissue images, for instance, to enable new fundamental research on inflammatory tissue remodeling. Moreover, the combination of this computational specificity with label‐free in vivo imaging via endomicroscopy would have great potential for clinical translation of this emerging technology.

Author Contributions

L.K. was involved in conceptualization, data curation, investigation, coding, validation, visualization writing – original draft, project management and funding acquisition. A.C. and M.R. were involved in conceptualization and coding. S.L. was involved in data curation. O.‐M.T. and B.C. were involved in investigation and validation, M.W., S.S., O.F., and R.H. were involved in writing – reviewing, project management and funding acquisition.

Funding

This work has received funding from the HORIZON EUROPE 2022 Marie Sklowska‐Curie Action under grant agreement 101103200 (to L.K.). The project was supported by the German Research Foundation (Deutsche Forschungsgemeinschaft, DFG) under project C01 of the collaborative research center TR‐R241 (to S.S. and M.W.).

Disclosure

The authors have nothing to report.

Conflicts of Interest

The authors declare no conflicts of interest.

Supporting information

Figure S1: Histograms of class labels and learning curves (a) for the binary classification of cells in mixture, (b) for the multi‐class classification from isolated cells using the original, imbalanced set, and (c) for the multi‐class classification of one exemplary, downsampled and balanced set.

Table S1: Model structure and number of trainable parameters (representation based on table 5.1 in Ref. [52] and adjusted for our data). * The classifier consisted of a dropout layer, a 2D conv layer, a sigmoid and an adaptive average pooling layer.

Table S2: Hyperparameters

Table S3: Per‐class performance metrics displayed as mean value within a 95% confidence interval (CI) from a 5‐fold cross validation. Please note that some folds are missing all class labels. Consequently, the CI for certain classes may appear wider, as their estimation is based on a reduced number of valid folds. This apparent inflation reflects the higher statistical uncertainty arising from missing class representations, rather than variability in model performance itself.

Data Availability Statement

The dataset used in this study is publicly available on Zenodo under a CC BY 4.0 license at https://doi.org/10.5281/zenodo.15657646. All code used for data preprocessing, model training, inference, and analysis for the binary and multiclass classification experiments is publicly available on GitHub at https://github.com/LucasKreiss/Multiphoton_Imaging_with_Computational_Specificity/tree/main/cell‐MICS under an MIT license.