Deep-learning-assisted snapshot optical tomography for microscopic volume prediction: a simulation study

Abstract

In this simulation study, we demonstrate fast-yet-accurate volume measurement of microscopic objects by combining snapshot optical tomography and deep learning. Snapshot optical tomography simultaneously collects a multitude of projection images, and thus can perform 3D imaging in a single snapshot. However, as with other wide-field microscopy techniques, it suffers from the missing-cone problem, which can seriously degrade the quality of 3D reconstruction. We use deep learning to generate a volume prediction from the 2D projection images bypassing the 3D reconstruction.

Introduction.

Despite recent progress in molecular assays and genetic testing, the assessment of structural changes in cells and tissues still serves as the gold standard for cancer diagnosis[1]. Among the observed morphological changes in tumor cells are alterations of nuclear structure[1]. As with the cell size, the nuclear size is regulated for cell homeostasis, leading to a constant ratio of nucleus to cellular volume in normal cells[2, 3]. Enlarged nuclei in cancer cells are related to aggressiveness and metastatic potential of cancer[4], and thus have been used to stage and prognose many different types of cancers[5–7]. Not surprisingly, a pathologist’s visual assessment of the size is prone to human error[8, 9], and the sampling number is often insufficient to detect rare cells. The 2D images used in the current practice do not correctly capture the 3D features either, which further contributes to a large variation in the size measurement. A method for high-throughput, automated measurement of the 3D size or volume can significantly improve the accuracy of the morphological evaluation.

For the cell volume measurement, a variety of methods are available, which include the Coulter method, optical microscopy, and atomic force microscopy[10]. The Coulter method measures the cell volume by monitoring the change in the impedance of an electroconductive-liquid-filled aperture as a cell passes through it[11]. The Coulter method is widely used in hematology and microbiology[10]; however, a particle’s shape and orientation can significantly affect the impedance measurement, up to 40%, as shown in Qin et al.[12]. In addition, the Coulter method and atomic force microscopy cannot be used to measure the volumes of intracellular organelles. Optical microscopy can overcome these challenges by measuring the thickness of a cell at every point of a 2D image. For the volume of an internal organelle such as nucleus, optical microscopy can determine the 3D boundary of an organelle, labeled with fluorescent dye, and calculate the volume by counting the number of voxels within the 3D boundary. For example, wide-field fluorescence microscopy in tandem with the objective-focus scan can record the boundary of an internal organelle in 3D and provide its volume. Wide-field fluorescence microscopy, however, suffers from the missing-cone problem, which seriously compromises the depth resolution. Without the missing-cone problem, confocal fluorescence microscopy can provide a higher depth resolution and is usually considered the gold standard for microscopic volume measurement[13]. However, the imaging throughput is not high, because, for volumetric imaging, a multitude of horizontal cross-sections need to be acquired while translating the sample along the optical axis direction.

Here we test a new type of volume measurement technique using snapshot optical tomography, which can capture the 3D image of a microscopic specimen in a single snapshot. With this technique, we can measure the volumes of fluorescently-labeled organelles in suspending cells at a high throughput without attaching them to a surface. We have demonstrated snapshot optical tomography using fluorescence[14], absorption[15], and refractive index[16] as imaging contrasts. As with wide-field microscopy, snapshot optical tomography suffers from the missing-cone problem, which can degrade the quality of tomographic reconstruction and seriously compromise the volume measurement accuracy. To overcome this limitation, we use deep learning, which predicts the volume directly from the projection images without a tomographic reconstruction. For the proof-of-concept demonstration, we use a database of 3D confocal images of cell nuclei[17] and simulate the projection images to be acquired with snapshot optical tomography. The simulated projection images are given as input to a multi-view convolutional neural network (MVCNN), which is trained with the true volume values obtained from the original confocal images. The volume measurement using snapshot optical tomography will be particularly useful for the diagnosis of hematologic malignancies (e.g., leukemia, multiple myeloma) or for the detection of circulating tumor cells, as it obviates the need for attaching the cells to a surface before interrogation. The MVCNN, which has inspired our work, was first demonstrated by Su et al.[18] for the recognition of 3D shapes from a series of 2D images recorded for varying camera orientations. For a thorough review of related research, we refer the readers to Yan et al.[19].

Dataset and data preprocessing.

We used the Statistics Online Computational Resource (SOCR) 3D Cell Morphometry Project Data[17], which includes 3D confocal images of 1552 nuclei for primary human fibroblast cells and human prostate cancer cells, labeled with DAPI (4’,6-diamidino-2-phenylindole). According to the published metadata, the 3D confocal images were acquired at the resolution of 0.132 μm and 0.315 μm along the transverse and axial directions, respectively. For each 3D data, we moved the nucleus to the center and resampled horizontal cross-sections at the grid spacing of 0.315 μm, the same as the axial resolution, using nearest-neighbor interpolation. Then, we axially padded the data with zeros so that the final size was 128×128×128. After randomization, we split 1552 nuclei data into 1242 training set, 155 validation set, and 155 test set. The true labels were calculated by counting the number of voxels within the nucleus volume and multiplying it with the volume for the voxel (0.0313 μm³). Figure 1(a) shows a 3D rendered image of an example cell nucleus included in the dataset. Figure 1(b) shows a histogram of the volume values for the entire data set, whose median volume is 493 μm³. Figure 1(c) shows a histogram of the thickness-to-width ratio, where the width is calculated from the projected area (A) onto a horizontal plane and given by $\sqrt{\left(4/\pi \right)A}$. The median value for the thickness-to-width ratio is 0.211. The wide distribution of the thickness-to-width ratio indicates that the volume estimation using a 2D image can produce large uncertainty.

Fig. 1.

Cell nucleus data used for this study: (a) an example of the nucleus volume, (b) the cell volume distribution, and (c) the thickness-to-width ratio distribution in the data set.

Snapshot optical tomography simulation.

For each nucleus volume, we generated 32 projection images, each of 128×128 pixels. To generate the projection images, we need to calculate the viewing direction of each lenslet. Figure 2(a) shows a schematic diagram of the imaging geometry used in snapshot optical tomography for fluorescence imaging. (X, Y, Z) are the Cartesian coordinates, and Z corresponds to the optical axis. The viewing direction for the projection image recorded with the (m, n)th lenslet is represented by a thick arrow, where m and n are the indices for the horizontal and vertical locations, respectively, of the lenslet in the array. The projection of the arrow onto the X-Z plane makes an angle of α_m with the Z axis, while the projection onto the Y-Z plane makes an angle of β_n. The two angle values uniquely define the viewing direction and can be calculated using Eqns. (1a) and (1b).

$${\alpha }_m={\sin }^{-1}\left\{\left(Mag/R{I}_0\right){\tan }^{-1}\left(\left(m-m_0\right)p/F_{TL}\right)\right\},$$ (1a)

$${\beta }_m={\sin }^{-1}\left\{\left(Mag/R{I}_0\right){\tan }^{-1}\left(\left(n-n_0\right)p/F_{TL}\right)\right\},$$ (1b)

where m = 1, 2, …, M, n = 1, 2, …, N, m₀ = (M + 1)/2, and n₀ = (M + 1)/2. For 6×6 projection images, M = N = 6, and m₀ = n₀ = 3.5. Mag is the magnification factor, RI₀ is the refractive index of the medium in which the sample is immersed, p is the width of each lenslet, and F_TL is the focal length of the tube lens.

Fig. 2.

Imaging geometry used for the simulation (a) and examples of the simulated images for a fluorescent bead before (b) and after (c) the reprojection operation. Scale bars in (b) and (c): 50 μm.

The projection image for the viewing direction (α_m, β_n) can be calculated using a Fourier slice theorem[20]. First, we calculate the 3D Fourier transform of the preprocessed, binary nucleus volume, then take a slice through the origin and perpendicular to the viewing direction. According to the original Fourier slice theorem, the 2D inverse Fourier transform of the slice provides the projection image for the viewing direction. Instead, we project the slice onto the U-V plane, where (U, V) are the spatial frequency components corresponding with the transverse coordinates (X, Y), then take the 2D inverse Fourier transform of the projected slice. This additional step is necessary because of different detector geometries. In the derivation of the original Fourier slice theorem, the detector rotates with the illumination so that the detector surface remains perpendicular to the viewing direction. For snapshot optical tomography, the detector is stationary and its surface is perpendicular to the optical axis irrespective of the viewing direction. For a more thorough explanation of the theory, we refer the readers to our previous work[14]. Figure 2(b) shows an example of the projection images typically acquired with snapshot optical tomography, which are simulated here using a spherical, fluorescent bead of the diameter 15 μm as a sample. Note that the projection images are elongated radially by different amounts depending on their positions in the 2D array. The projection images at the corners are not usually recorded, as their viewing angles are greater than the maximum angle allowed by the numerical aperture of the objective lens. The raw image comprising 32 projection images can be directly used in a tomographic reconstruction algorithm to provide a 3D distribution of fluorophores in the bead[14]. Instead, we reproject each projection image onto a plane perpendicular to the viewing direction, which removes the elongation effect. The projection images without the elongation, shown as in Fig. 2(c), are given as input to the MVCNN for a volume prediction. Without the elongation effect, the projection images are identical to each other for a spherical bead. For small organelles, the diffraction effect becomes dominant, which would require a reprojection operation using the scalar diffraction theory[21].

MVCNN for volume prediction from projection images.

We have constructed a MVCNN comprising two blocks, as shown in Fig. 3(a). The first block consists of 32 parallel subnetworks, each of which is a 2D convolutional neural network (CNN1) extracting a multitude of features from one of the projection images. The outputs of the subnetworks are concatenated, then given as an input to the second block, which is also a 2D convolutional neural network (CNN2) predicting the volume of the imaged specimen. Figure 3(b) shows the architecture of CNN1, which contains two Conv2D layers, each of which is followed by the AveragePooling2D layer, then the Flatten layer, and finally the Dense layer. Each Conv2D layer employs a 4×4 kernel that filters through the projection image of 128×128 pixels to extract key features. It applies the rectified linear unit (ReLU) activation function and the Glorot uniform initialization with a different seed number for each hidden layer. The AveragePooling2D layer with the 4×4 pool size is used to obtain the features proportional to the area of the projection image. The Flatten layer after the second AveragePooling2D layer converts the output tensor to a 1D array. The Dense layer of 16 nodes summarizes the features extracted from each projection image. After concatenating the outputs to produce 32×16 features in total, we append four zero tensors, each with 1×16 size. The outputs of the first block are reshaped to 6×6×16, where the last dimension represents the number of features. Figure 3(c) shows the architecture of CNN2, which contains a similar structure to CNN1 except for the following changes. First, as the input size is significantly smaller, the kernel size of the Conv2D layers is reduced from 4×4 to 2×2. Second, the AveragePooling2D layers are replaced with the MaxPooling2D layers, which identify the features that are most descriptive of the predicted volume. Third, a Dropout layer is added to alleviate the overfitting problem, and the final Dense layer is added for the volume prediction. We use L2 regularization for the Conv2D layers in CNN2 and the mean squared error (MSE) for the loss function.

Fig. 3.

MVCNN architecture used in this study. (a) shows the structure for the entire MVCNN, while (b) and (c) show the structures for CNN1 and CNN2, respectively.

Figure 4 depicts the three input types that were used for the MVCNN analysis: (1) 32 area values calculated from the projection images, (2) 32 binarized projection images, and (3) 32 projection images with the intensity normalized by the maximum within each projection image. The numbers in Fig. 4(a) are the area values, calculated by counting the number of pixels of the nucleus region in each projection image. Figure 4(b) depicts the shadows of the cell nuclei, obtained by thresholding the projection images. The cell nuclei hold a value of 1 and the background a value of 0. Figure 4(c) shows the 32 projection images after the normalization. The background is denoted with a value of 0, while the cell nuclei vary with intensity up to a maximum of 1. The normalization is necessary because the fluorescence signal depends on the intensity and uniformity of the excitation light, which are subject to change for different instruments, as well as the labeling efficiency and quantum yield of the fluorophores, which are expected to vary widely. For the first input type, we bypass the projection image processing and directly provide the area values to the second block of the MVCNN (i.e., CNN2), which accepts 6×6×1 tensors as inputs. To train the neural network, we used the learning rate of 10⁻⁴, the early stopping with the patience of 100, and the batch size of 20. The training was completed in 3 min on a laptop (Intel i7-10870H CPU, 32 GB memory) with one general-purpose graphics processing unit (NVIDIA, GeForce RTX 3080 Laptop GPU). For the second and third input types, we used the full MVCNN with CNN2 modified to accept 6×6×16 tensors as inputs. To train the neural network, we used the learning rate of 10⁻⁵, the early stopping with the patience of 50, and the batch size of 20. The training took about four days on the same laptop. The training could be expedited using a higher learning rate (about 3 hours with the learning rate of 10⁻⁴), although it introduced unstable convergence for some cases. Once the MVCNN was trained, it predicted the volume immediately (about 0.06 s) for the test set.

Fig. 4.

Examples of the three input types tested with the MVCNN: (a) area values, (b) binarized projection images, and (c) normalized projection images.

Figure 5(a) represents an example of the predicted volume from the MVCNN in comparison to the label—the true volume generated from the SOCR 3D Cell Morphometry Project Data. The data fits to a line (shown in red) of the slope 0.998±0.009 and the intercept −5.32±5.87 with the adjusted R-squared value of 0.989. For comparison, Fig. 5(b) shows the volume plotted with the projected area for the same dataset. When a single 2D image from wide-field imaging is used, the projected area or a metric derived from it would be used to estimate the nucleus volume. The adjusted R-squared value for the linear regression is only 0.186, which emphasizes the importance of the volume measurement using 3D volumetric imaging instead of 2D imaging. As the total number of data was only 1552, we have performed five independent runs of training and testing, each time using new randomly-split subsets of the SOCR 3D Cell Morphometry Project Data and training from scratch. The accuracy of the MVCNN was evaluated using the average of the five results.

Fig. 5.

Prediction of the nucleus volume using the test set formatted in the third input type: (a) an example volume prediction using the MVCNN, and (b) the same volume data plotted with the projected area values.

Table 1 shows the mean absolute error (MAE) of five runs through the MVCNN for the test set formatted in three different input types. The area value data type is shown to produce the MAE of 6.97±0.91% (sample number, 5), which is quite low (i.e., the accuracy is high) by using the area values from the multi-view projection images, instead of the area for a single view. Using the binary images, the MVCNN produces a similar MAE of 6.47±0.93% (sample number, 5), which implies the first block of the MVCNN was properly trained to obtain the area of the nucleus in each projection image. Using the normalized images, the MAE further decreases to 3.89±0.31% (sample number, 5). The small MAE value may be attributed to the depth information encoded in the intensity variation in each projection image—the information not available in the binarized projection images. For all the three data types, the standard deviations are high because the amount of data available for training and testing was not large. After careful tuning of the hyperparameters, the difference betwen the training and testing accuracies was less than 1%.

Table 1.

Mean absolute error (%) of MVCNN for a volume prediction using the test set formatted in three different types.

	1	2	3	4	5	Summary
Area	7.30	5.65	6.42	7.73	7.74	6.97±0.91
Binary	6.59	5.33	6.10	6.45	7.88	6.47±0.93
Normalized	3.62	3.52	4.25	3.99	4.09	3.89±0.31

Although any optical imaging techniques with a depth-sectioning capability can provide the volume of an organelle, the accuracy of volume prediction is highly variable due to different imaging resolutions, the depth resolution in particular. Many optical microscopy techniques including wide-field fluorescence microscopy and differential interference contrast microscopy suffer from the missing-cone problem[22], which significantly compromises the depth resolution. The missing cone refers to the empty region resembling an apple core in the 3D Fourier space, which occurs because the objective lens cannot capture all the light rays emitted from fluorophores. A well-known artefact due to the missing cone is the smoothing of the reconstructed object along the optical axis direction (i.e., the thickness direction), which significantly lowers the accuracy of volume prediction[23]. Deconvolution using the 3D point spread function for the imaging system helps alleviate the problem[24]. However, optimal choices for hyperparameters (e.g., the number of iterations) can vary depending on the shape of the imaged object, which is not known a priori. Even with cutting-edge deconvolution algorithms, the performance of volume prediction rapidly degrades as the ratio of thickness to lateral width of the sample decreases, when a technique with the missing-cone problem is used. Confocal microscopy is free from the missing-cone problem and can measure the volume at high accuracy; however, the imaging speed is very low, and it requires the sample to be stationary during the data acquisition. Another idea to alleviate the missing-cone problem is to acquire a multitude of projection images while the sample is rotated in a microfluidic channel[25]. Snapshot optical tomography allows for snapshot 3D imaging, and thus can be applied to suspending cells as well as adherent cells. Although it suffers from the missing-cone problem, we have shown that deep learning can still provide high accuracy of less than 4% error for the volume prediction.

For the experimental demonstration, confocal microscopy and snapshot optical tomography will have to be applied to the same cells. In this study, to train and test the neural network, we used about 1500 nuclei volumes. Collecting about the same amount of data using both confocal microscopy and snapshot optical tomography is not trivial but can be done. Once the neural network is trained, the nucleus volumes can be directly obtained from the snapshot optical tomography data, which can be acquired at high throughput. As we normalized the projection images, in which the variations due to the excitation light or fluorophores have been removed, the same neural network can be used for any samples and fluorophores. Combining snapshot optical tomography with deep learning, the developed method will allow for measuring the volumes of intracellular organelles at high throughput and accuracy. With multiple fluorophores of different colors, the same method can be used to acquire various morphometric parameters such as the nucleus-to-cytoplasmic ratio and mitochondria-to-cytoplasm ratio, using the full 3D information instead of a 2D cross section. The deep-learning-assisted snapshot optical tomography would also be useful where the volume or shape of microparticles needs to be acquired at high throughput and accuracy, for example, microplastics in drinking water and pharmaceutical powders.