Single Molecule Localization Super-resolution Dataset for Deep Learning with Paired Low-resolution Images

Abstract

Deep learning super-resolution microscopy has advanced rapidly in recent years. Super-resolution images acquired by single molecule localization microscopy (SMLM) are ideal sources for high-quality datasets. However, the scarcity of public datasets limits the development of deep learning methods. Here, we describe a biological image dataset, DL-SMLM, which provides paired low-resolution fluorescence images and super-resolution SMLM data for training super-resolution models. DL-SMLM consists of six different subcellular structures, including microtubules, lumen and membrane of endoplasmic reticulum (ER), Clathrin coated pits (CCPs), outer membrane of mitochondria (OMM) and inner membrane of mitochondria (IMM). There are 188 sets of raw SMLM data and 100 signal levels for each low-resolution image. This allows software developers to generate thousands of training pairs through data segmentation. The performance of the imaging system was further evaluated using DNA origami samples. Finally, we demonstrated examples of super-resolution models trained using data from DL-SMLM, highlighting the effectiveness of DL-SMLM for developing deep learning super-resolution microscopy.

Background & Summary

Fluorescence microscopy is an indispensable tool for biological research. However, the resolution of conventional optical microscopy was restricted by diffraction limit (Fig. 1a). Super-resolution microscopy has rapidly advanced since the late 20th century and now plays a crucial role in today’s research. Mainstream super-resolution technologies include structured illumination microscopy (SIM)¹, stimulated emission depletion (STED)^2,3 microscopy and single molecule localization microscopy (SMLM, Fig. 1b)^4–6, breaking the diffraction limit of optical systems through unique hardware designs. Meanwhile, software-based techniques for achieving super-resolution imaging have also been developed^7,8. These methods aim to achieve super-resolution imaging via computational method, rather than relying on expensive customized hardware. In particular, recently, the rapid advancement of deep learning technology has spurred extensive research into using deep learning models to enhance microscope images.

Fig. 1.

Principle of deep learning super-resolution microscopy. a. Optical diffraction leads to overlapping signals from adjacent fluorescent molecules, which limits the resolution of fluorescence images. b. SMLM achieves super-resolution imaging by localizing the coordinates of independent single molecules. c. Principle of deep learning super-resolution microscopy. The super-resolution model is trained using LR images as input and corresponding super-resolution images as labels (such as SMLM images). After training, the model can be applied to enhance the resolution of previously unseen diffraction-limited images.

Deep learning based single image super-resolution task aims to convert a low-resolution (LR) image into a high-resolution (HR) image through computational reconstruction. This depends on deep learning models to solve the inverse problem in super-resolution tasks^9,10. Generally, super-resolution models for microscopy are trained on a dataset that contains LR and HR image pairs. Images captured from conventional fluorescence microscopy are typically used as LR data, and super-resolution images of the same regions reconstructed using SMLM can be used as corresponding HR labels (Fig. 1c). Obviously, high-quality HR labels are the foundation for improving the performance of super-resolution models. For example, based on the SIM dataset BioSR¹¹, several superior SIM reconstruction algorithms have been proposed^12–14, facilitating biologists’ research into the rapid dynamic temporal changes in live cells.

However, scarce data resources still pose an obstacle to the development of deep learning super-resolution microscopy. Super-resolution images acquired by SMLM have the highest spatial resolution among all super-resolution techniques, which makes them ideal sources for super-resolution image datasets. For example, trained on paired LR-SMLM data, single-frame super-resolution microscopy (SFSRM) achieves nanoscale spatial resolution with millisecond temporal resolution¹⁵. This capability allows biologists to image dynamic processes occurring at nanoscale within cells, providing insights into temporal changes and interactions at high resolution.

Unfortunately, because the acquisition of high-quality SMLM data is difficult and time-consuming, there is a shortage of open source SMLM dataset, which limits research and development in this area. SMLM imaging needs to optimize many important aspects, including biological samples, fluorescence labels, microscope system, and localization algorithm. Additionally, thousands of frames must be accumulated, which may take hours for a single SMLM image and restrict the production of SMLM data.

Here, we provide a dataset, DL-SMLM¹⁶, which is aimed for the training of deep learning super-resolution microscopy, specifically methods enabled by SMLM. DL-SMLM contains aligned LR and HR image pairs of subcellular structures in fixed cells, acquired from total internal reflection fluorescence (TIRF) microscopy¹⁷ and SMLM, respectively. This dataset consists of six biological structures: Clathrin coated pits (CCPs), microtubules, Endoplasmic reticulum (ER) lumen, ER membrane, outer membrane of mitochondria (OMM) and inner membrane of mitochondria (IMM), as shown in Fig. 2a. The original field of view (FOV) for LR and HR images is 33.28 μm × 33.28 μm, and zoomed-in regions of 5.12 μm × 5.12 μm are also shown. The hollow structure between two ER membranes can be resolved with a width of 81.25 ± 16.25 nm (Fig. 2b,c), and the diameter of a CCP is measured as 195 nm (Fig. 2d,e).

Fig. 2.

Visualization of super-resolution images in DL-SMLM. a. Six different subcellular structures and zoomed-in regions are shown, including TIRF and SMLM images. b. Zoomed-in region of the ER membrane indicated by the red box in a. c. Intensity profiles of the ER membrane indicated by the orange boxes in b, respectively. The numbers indicate distance between peaks. d. Zoomed-in region of the CCPs indicated by the red box in a. e. Intensity profiles of the CCPs indicated by the orange arrows in d, respectively. Scale bar, 5 μm and 1 μm (zoomed-in regions) in a, 100 nm in b, 200 nm in d.

For each LR image data, we provide a 100-frame raw image stack with exposure time of 50 ms, which allows users to generate LR images with different signal-to-noise ratios (SNR) by combining different numbers of frames together. HR images have a pixel size of 16.25 nm, representing an 8-fold resolution enhancement compared to the LR images, which has a pixel size of 130 nm. Raw localization data is also uploaded to the dataset to allow users to generate customized SMLM reconstructions with different scales. LR and HR images were acquired in the same region of interest (ROI) with real-time drifting correction system (described below in the optical setup section), the focus system maintains ROI fixed throughout the entire imaging process, thereby achieving nanometer-level accurate alignment.

To validate the quality and utility of the dataset, we trained the SFSRM model on DL-SMLM. In the test data, the output of the super-resolution model exhibits high structural consistency with the genuine SMLM reference (Fig. 5), indicating reliable performance and accuracy of the model. We further validated the performance of the super-resolution model in live cell imaging (Fig. 6). In the imaging of the OMM and ER membrane, morphological changes in the structure can be observed, which are not discernible in LR images. While these biological phenomena were not further explored in this work, the examples demonstrate the significant potential of DL-SMLM for analysing dynamic processes in live cells imaging.

Fig. 5.

Test images reconstructed using the SFSRM model trained on DL-SMLM. SFSRM models were trained and tested using DL-SMLM data. Microtubules: 17 training images, 3 testing images. ER membrane and OMM: 26 training images, 4 testing images. Representative test images are shown. Merged images show the overlap between SFSRM reconstructions (green) and ground truth SMLM images (red). Scale bar, 5 μm and 0.5 μm (zoomed-in regions).

Fig. 6.

Live-cell images reconstructed using the SFSRM model trained on DL-SMLM. a and c. OMM and ER membrane of COS-7 cells were labelled through transfection of TOMM20-HaloTag and mScarlet3-SEC. 61β for live cell imaging, and super-resolution was performed by SFSRM. b. Time-lapse zoomed-in images of the OMM, as indicated by the white box in a. Protrusions are found on the OMM, and these structures are observed to extend and move downward, as indicated by the white arrow in b. d. Time-lapse zoomed-in images of the ER membrane, as indicated by the white box in c. Time-lapse imaging demonstrates that the ER membranes undergo a dynamic process where they first approach each other and then quickly separate, as indicated by the white arrow in d. Scale bar, 5 μm in a and c, 0.5 μm in b and d.

In conclusion, DL-SMLM exhibits high-quality data, enabling the training of highly performant super-resolution models. This capability is of significant importance to the development of deep learning super-resolution microscopy.

Methods

Optical setup

All the data was acquired using a TIRF microscope built based on Nikon Ti2-U inverted fluorescence microscope as shown in Fig. 3a. Lasers of 488 nm (PIC, 150 mW), 561 nm (Coherent, 150 mW) and 639 nm (CNI, 400 mW) were coupled to an optical fiber and then were collimated by a lens. Collimated illumination beam was off axis focused on the back focal plane (BFP) of objective to generate TIRF. The microscope was equipped with a high numerical aperture (N.A.) oil-immersion objective (Plan Apo 100 × , N.A. = 1.45) in order to improve imaging quality. A 4-band filter set (Chroma, TRF89901-EMv2) was used to collect fluorescent signals from different channels. An sCMOS camera (Hamamatsu, ORCA-Fusion BT) is used for recording. During the image acquisition, we set a ROI range of 256 × 256 pixels and binning of 2, resulting in pixel size of 130 nm and FOV of 33.28 μm × 33.28 μm.

Fig. 3.

Optical setup and imaging of standard DNA origami samples. a. The illumination path used a common TIRF mode. Fluorescent signals were collected by a 100 × high N.A. (1.45) objective and finally projected onto the sCMOS. Drifting was corrected in real time by a PIEZO stage according to images captured by an infrared camera. DM: dichroic mirror. b. 20 nm and 10 nm DNA origami structures can be resolved by DNA-PAINT imaging. c and d. Intensity profiles of the orange boxes in b, respectively. The numbers indicate distance between peaks. Scale bar, 200 nm and 50 nm (zoomed-in region) in b.

The real-time lock-in focus system ensures precise alignment and registration between LR images and SMLM images, using a layer of polystyrene microspheres (Huakeweike, PS-M-10075) with diameter of 4.46 μm placed on the coverslips before seeding cells. An infrared LED and an infrared camera were used to monitor the 3D position of these microspheres. Drifting of samples could be corrected in real-time by a PIEZO stage with nanometre precision, which can maintain the sample position stable over hours during the entire image acquisition, with the minimum measured standard deviation (std.) approximately of 0.736 nm in the x direction, 0.857 nm in the y direction and 0.474 nm in the z direction.

Sample preparation

Cell culture

COS-7 cells (3111C0001CCC000033, National Infrastructure of Cell Line Resources, China) were grown in Dulbecco’s modified Eagle’s medium (DMEM, Gibco, c11995500BT) supplemented with 10% fetal bovine serum (Gibco, 16000-044) and 100 U/ml penicillin and streptomycin (Gibco, 15140122) at 37 °C and 5% CO₂ condition. Cells were digested by trypsin (Gibco, 25300062) and passaged every 2-3 days.

Immunofluorescence labelling

SMLM imaging was performed using two established methods: stochastic optical reconstruction microscopy (STORM)⁴ and DNA points accumulation for imaging in nanoscale topography (DNA-PAINT)^18,19. Specially, images of CCPs and OMM were collected by STORM, and images of microtubules, ER lumen, ER membrane and IMM were collected by DNA-PAINT, as shown in Table 1.

Table 1.

Fluorescence labelling technique used for SMLM super-resolution imaging.

Samples	Imaging methods	Labelling methods	Fluorescence probes	Quantities
CCPs	STORM	Anti-Clathrin heavy chain antibody	AF647(2nd antibody)	28
ER lumen	DNA-PAINT	Anti-GFP nanobody	Cy3B(imager)	26
ER membrane				30
Microtubules				33
OMM	STORM	Anti-TOMM20 antibody	AF647(2nd antibody)	30
IMM	DNA-PAINT	Anti-ALFA nanobody	Cy3B(imager)	41
				Totally 188

For sample preparation of STORM, about 50,000 COS-7 cells were seeded in 35 mm glass-bottomed dishes (Cellvis, D35-14-1-N) and cultured for 24 hours before fixation with 3% paraformaldehyde (Electron Microscopy Sciences, 15710) and 0.1% glutaraldehyde (Electron Microscopy Sciences, 16200) in Phosphate Buffered Saline (PBS, Sigma-Aldrich, P4417) for 10 min at room temperature. Excessive fixation buffer was reduced by 100 mM NH₄Cl (Sigma-Aldrich, A9434) for 7 min. Then cells were washed with PBS and were permeabilized with 0.2% Triton X-100 (Sigma-Aldrich, 93443) for 7 min. After rinsed with PBS, cells were blocked for 90 min using 10% Normal Goat Serum (NGS, Beyotime, C0265) and 0.05% Triton X-100 in PBS and then stained with the primary antibodies for 60 min in antibody dilution buffer (5% NGS, 0.05% Triton X-100 in PBS). Cells were washed with washing buffer (1% NGS, 0.05% Triton X-100 in PBS) five times for 15 min per wash, followed by incubation of the secondary antibodies for 60 min. Cells were then washed with washing buffer five times for 15 min per wash and then washed with PBS for 5 min. After that, cells were post-fixed with 3% paraformaldehyde and 0.1% glutaraldehyde in PBS for 10 min. Next, cells were washed three times with PBS, 5 min per wash and then washed twice with ddH₂O, 3 min per wash. Finally, the cells were stored with ddH₂O at 4 °C. For mitochondria samples, anti-TOMM20 primary antibody (Abcam, ab186735) was used. For CCPs samples, anti-Clathrin heavy chain primary antibody (Abcam, ab21679) was used. Both the primary antibodies were diluted to 1:200 and goat anti rabbit IgG Alexa Flour 647 secondary antibodies (Invitrogen, A21245) were diluted to 1:200 for staining.

For DNA-PAINT samples, about 50,000 COS-7 cells were also seeded in 35 mm glass-bottomed dishes and cultured for 16 hours. Plasmids of MAP7-3 × GFP, EGFP-KDEL, EGFP-SEC. 61β and cox8a-mStayGold-ALFAtag were transfected into COS-7 cells by Lipofectamine 3000 reagent (Invitrogen, L3000075) in order to label microtubules, ER lumen, ER membrane and IMM, respectively. After about 24 hours, the cells were fixed, reduced and permeabilized as described above. Then, cells were blocked for 90 min with antibody incubation buffer (Massive Photonics). Anti-GFP nanobody and anti-ALFAtag nanobody conjugated with DNA docking strands (Massive Photonics) were diluted to 1:100 in antibody incubation buffer and incubated for 60 min. After that, cells were washed with washing buffer (Massive Photonics) three times for 10 min per wash and washed twice with PBS. Finally, the cells were stored with ddH₂O at 4 °C.

Live cell samples

About 50,000 COS-7 cells were seeded in 35 mm glass-bottomed dishes for 24 hours before transfection experiment. Plasmids of TOMM20-HaloTag and mScarlet3-SEC. 61β were transfected into cells by Lipofectamine 3000 to label OMM and ER membrane, respectively. Culture medium was replaced after transfection for 8 hours and cells were grown for 24 hours before imaging. For TOMM20-HaloTag transfection, Janelia Fluor 549 HaloTag Ligand (Promega, GA1110) was incubated for 20 min at 37 °C before imaging.

Data acquisition

Before STORM imaging, samples were sealed with oxygen-depleted STORM imaging buffer, which consisted of 1 × PBS buffer, 10% glucose, GLOX (glucose oxidase (0.6 mg/ml) and catalase (0.06 mg/ml), dissolved in Tris-HCl buffer) and 143 mM 2-mercaptoethanol. Glucose (101141632), glucose oxidase (G2133), catalase (C1345) and 2-mercaptoethanol (M3148) were ordered from Sigma-Aldrich. To avoid photobleaching before STORM imaging, 647 nm laser was set to low power to search and focus the target cell. The size of ROI was set to 256 × 256 pixels as described above. The drift correcting system began to run when the target cell was chosen and focused perfectly. First the LR image of target region was collected with low excitation intensity of ~20 mW·cm⁻² and 50 ms exposure time, 100 frames totally. Then STORM imaging was carried out with high excitation intensity ~3 kW·cm⁻². Exposure time was still 50 ms and 30,000 frames were acquired totally.

For DNA-PAINT imaging, cells were washed with imaging buffer before imaging. Then Cy3B imager buffer was diluted in imaging buffer to a concentration of 1 nM and added to cells. The imaging buffer and imager were purchased from Massive Photonics. Same as the procedure of STORM imaging, first the LR images were acquired. 488 nm laser was used to image GFP or mStayGold and search and focus target region. When the target region was chosen and focused perfectly, the drift correcting system began to run. LR image of 100-frame was collected with low excitation intensity of ~20 mW·cm⁻² and 50 ms exposure time. Then 561 nm laser was applied with high excitation intensity ~600 W·cm⁻² to perform DNA-PAINT imaging. 20,000 or 30,000 frames were acquired for the final reconstruction and exposure time was set to 100 ms per frame.

During the live cell imaging, 561 nm laser was used to excite JF549 or mScarlet3 with low excitation intensity ~2 mW·cm⁻² to reduce photobleaching and phototoxicity. 75-second or 100-second videos were collected with the exposure time of 50 ms.

Data processing

Spot detection and fit

The single molecule spots of raw data were identified and fitted by ‘Picasso: Localize’¹⁹. The parameters were set as described below. For identification, box side length was set to 7 and min net gradient was determined by raw data to identify spots as many as possible and avoid incorrect identification meanwhile. For photon conversion, parameters were set according to the camera. The EM gain was set to 1 to minimize the impact of the sCMOS camera, as the software was originally designed for EMCCD cameras. Baseline was 100, and sensitivity (conversion factor electrons per count) was 0.23, and quantum efficiency was 0.9, and pixel size was set to 130 nm. As for fit methods, we prefer maximum likelihood estimation (MLE) instead of least squares (LQ), because the former can obtain more accurate localizations.

HR image reconstruction

The localization data files of ‘hdf5’ format carried out by ‘Picasso: Localize’ were opened by Render of Picasso and exported as ‘csv’ files, which could be opened by ThunderSTORM²⁰, an ImageJ plugin. Single molecule spots with abnormal photon number (less than 100 or more than 100,000) were filtered out. HR images were reconstructed with magnification of 8 using Histograms method, which allowed no extra post processing such as Gaussian blur. Users can also reconstruct HR images with other magnification and methods, as we provide both original single molecule localization data and reconstructed HR images in DL-SMLM.

Data Records

DL-SMLM is available on Figshare¹⁶ at 10.6084/m9.figshare.26879218.v1 under CC-BY license. The images are in tiff format and single molecule localization data is in csv format. The following files are provided:

CCPs.zip: providing 28 sets of data of CCPs.

Microtubules.zip: providing 33 sets of data of microtubules.

ER-KDEL.zip: providing 26 sets of data of ER lumen.

ER-sec. 61β.zip: providing 30 sets of data of ER membrane.

OMM.zip: providing 30 sets of data of OMM.

IMM.zip: providing 41 sets of data of IMM.

Each set of data contains a 100-frame LR image stack, a sum LR image of 100 frames, a reconstructed SR image with magnification of 8 and raw single molecule localization data. Detailed labelling information of DL-SMLM can be found in Table 1.

Technical Validation

DNA origami imaging

DNA origami imaging is a powerful tool to validate SMLM performance²¹. Here, we carried out DNA-PAINT imaging of origami to demonstrate resolution and stability of our optical system. 20 nm and 10 nm grid DNA origami was synthesized according to previous studies¹⁹. Origami was laid on coverslips with 4.46 μm polystyrene microspheres, and then DNA-PAINT imaging was carried out using Cy3B imager of 1 nM concentration. There were 30,000 frames collected totally. After spot identification, fitting, reconstruction and redundant correlation correction (RCC), 20 nm and 10 nm grid origami structures were resolved by DNA-PAINT imaging (Fig. 3b) with a mean photon number of 88,044 and mean localization precision of 1.04 nm. The distance between adjacent spots for 20 nm grid origami is 20.20 ± 0.79 nm and 10.10 ± 1.63 nm for 10 nm grid origami (Fig. 3c,d).

Analysis of resolution

The localization precision is mainly dependent on the number of detected photons²². In Fig. 4a, we displayed photon contributions of HR images shown in Fig. 2a. The distribution of localization precision shows a mean of 4–6 nm (Fig. 4b), which was calculated by QC-STORM²³.

Fig. 4.

Resolution analysis of super-resolution data in DL-SMLM. a. The photon counts distributions of SMLM images in Fig. 2a show the mean photon number of each image. b. The distributions of localization precision show the mean of each image.

Network training

For microtubules, 17 images from DL-SMLM dataset were designated for training and the remaining 3 images reserved for testing. For ER membrane, 26 images from the dataset were used for training and the other 4 images for testing. For OMM, 26 images from the dataset were used for training and the other 4 images for testing.

Model training and inference were performed on a computer workstation with an i7-9700 CPU running at 3.00 GHz (Intel) and an RTX 3090 graphics card (NVIDIA). The processes were conducted via Python v3.9 and PyTorch v2.2.2. Adam was used for stochastic optimization, and the random number of seeds was set to 20.

The SFSRM model was trained according to previously reported methods¹⁵. We used fixed cell data from DL-SMLM as training data, augmented by random cropping, rotation, and flipping. SFSRM employs LR images and their edge gradient maps as dual-channel inputs, which are reconstructed using the NanoJ-SRRF plugin in ImageJ, with the ring radius set to 1 and the number of axes per ring set to 5. The loss function in SFSRM training is divided into three stages, gradually transitioning from pixel-wise loss functions to non-pixel-wise loss functions, as shown in the following formulas.

$${\mathrm{SFSRM}}_{\mathrm{\theta }}=\underset{\mathrm{\theta }}{\mathrm{argmin}}\left\{\begin{array}{ll}{\mathrm{L}}_{{\mathrm{MSSSIM}}_{\mathrm{L}1}}& \left(n\in [1,\mathrm{n}1\right]\\ {\mathrm{L}}_{{\mathrm{MSSSIM}}_{\mathrm{L}1}}+\mathrm{\delta }\cdot {\mathrm{L}}_{\mathrm{Percep}}& n\in \left[\mathrm{n}1,\mathrm{n}2\right]\\ {\mathrm{L}}_{{\mathrm{MSSSIM}}_{\mathrm{L}1}}+\mathrm{\delta }\cdot {\mathrm{L}}_{\text{Percep}}+\mathrm{\beta }\cdot {\mathrm{L}}_{\mathrm{Adv}}+\mathrm{\gamma }\cdot {\mathrm{L}}_{\mathrm{Freq}}& n\in \left[\mathrm{n}2,\mathrm{n}3\right]\end{array}\right)$$

In this setup, n represents the total number of iterations, while n1, n2, and n3 denote the iteration counts that partition the training process into distinct training stages, specifically set at n1 = 10,000, n2 = 15,000, and n3 = 20,000. The balance weights were set to δ = 0.1, β = 0.001 and γ = 0.01. The loss function ${\mathrm{L}}_{\mathrm{MSSSIM\_L}1}$ combines multiscale structure similarity loss (MS-SSIM)²⁴ and L1 norm losses, L_Percep refers to perceptual loss derived from a pretrained VGG network²⁵, L_Adv indicates the adversarial loss from a GAN framework²⁶, and L_Freq measures frequency loss in Fourier space; further details on the SFSRM method can be found in its description. The learning rates of the generator model were set to 3 × 10⁻⁴, 1 × 10⁻⁴ and 5 × 10⁻⁵ for each phase of training. The learning rate of the discriminator was set to 1 × 10⁻⁵. The training of the SFSRM took approximately 20 hours with a bath size of 1.

After training of models, live cell image stacks of OMM and ER membrane were used as the input of models to predict HR images. As shown in Fig. 6, reshaping of OMM and ER membrane can be observed in predicted HR images, which can hardly be distinguished in the LR images.

Usage Notes

Train data and test data for deep learning can be generated from raw data of DL-SMLM by segmentation and augmentation, just like common processes in deep learning. Customized reconstructions and post processing methods can be applied according to users’ purposes. We suggest magnification of no more than 13 according to localization precision. DL-SMLM can also be used for resolution analysis, localization precision analysis, or for other potential applications.

Author contributions

W.J., T.X. and L.G. designed the experiments. X.Z. developed the imaging system. X.Z. and T.Y. performed data analysis. X.Z. and T.P. conducted sample preparation and imaging. X.Z., T.Y. and T.P. wrote the manuscript, which was modified by all of the other authors.

Code availability

All the code or software for generation or processing of the dataset has been detailed in the Methods section. ‘Picasso’ for DNA-PAINT data processing is available via GitHub at https://github.com/jungmannlab/picasso. For additional information, please contact the authors.

Competing interests

The authors declare no competing interests.