3D localization of retrovirus assembly in the presence of structured background with deep learning

Abstract

Human immunodeficiency virus type 1 (HIV-1) particle assembly is driven by the Gag structural polyprotein and is a crucial step in the production of new virus particles. Elucidating the details of this process is necessary to fully understand the virus replication cycle. Real-time measurements of virus particle biogenesis in living cells have proved challenging, and most of our knowledge of this process to date has come from total internal fluorescence microscopy of labeled Gag at the bottom plasma membrane (PM) of adherent cells. While the glass coverslip adjacent to the bottom PM renders this an artificial environment, fluorescence measurements at the more physiologically relevant top PM are challenging due to the three-dimensional (3D) profile at the top PM as well as the large, structured background fluorescence that arises due to cytoplasmic, unassembled Gag protein. Here, we describe an approach to 3D localization microscopy and analysis to address the challenges associated with imaging virus assembly at the top PM in live cells. Specifically, we have employed the double helix point spread function for 3D imaging with an extended depth of field combined with a deep learning pipeline to analyze images that contain heterogeneous structured backgrounds. We demonstrate the power of this approach by measuring virus assembly at the top PM of adherent cells in 3D fluorescence microscopy and observe intriguing differences in the assembly kinetics and HIV-1 Gag puncta mobility between the adherent bottom PM and the nonadherent top PM.

INTRODUCTION

The assembly of human immunodeficiency virus type 1 (HIV-1) particles is a complex process involving interactions of retroviral structural proteins with numerous cellular factors. Many gaps in our knowledge of this process remain to be addressed. The main driver of retrovirus particle assembly is the Gag polyprotein, which when expressed in cells triggers the assembly and release of immature particles (1). Gag is expressed in the cytoplasm and forms high-order oligomers at particle assembly sites on the inner leaflet of the plasma membrane (PM). Live-cell fluorescence imaging of labeled Gag into punctate assemblies has allowed the kinetics of the assembly process to be measured at a single-assembly level. Because the cytoplasmic population of unassembled HIV-1 Gag appears as a large, structured background signal in fluorescence microscopy, monitoring HIV-1 assembly typically utilizes total internal reflection fluorescence (TIRF) microscopy to suppress the background signal from the cytoplasm to identify Gag puncta biogenesis in the vicinity of the glass-cell interface (2–4).

While TIRF microscopy significantly enhances the signal/background ratio, it is inherently limited to imaging of the bottom PM of adherent cells (5). Thus, the release of virus particles observed in TIRF microscopy occurs next to the glass interface, which confines the released particles to the small space between the cell and the coverslip (2,3). Currently, it is unknown whether the presence of a glass coverslip adjacent to the bottom PM affects the virus particle biogenesis at these locations. However, it has been reported that the presence of a hard glass coverslip affects the properties of the PM that may be expected to affect viral particle assembly and release. For example, the mobility of certain PM proteins has been reported to be significantly lower on PM adjacent to glass than nonadherent PM (6). The hardness of the substrate has also been reported to affect actin cytoskeletal structure and rheology, with hard substrates leading to more stress cables and more (nematic) alignment of the actin cytoskeleton and a more solid-like structure; softer substrates have been associated with fewer stress cables, more isotropic organization, and fluid-like flow of actin (7–9). Distinctive actin structures have been seen at retrovirus assembly sites, and the Gag protein was observed to preferentially nucleate in regions of low F-actin density (10–12). Due to these and other observations implicating the association of Gag with actin, observation of the virus assembly process at PM locations that are not in contact with the glass coverslip is warranted.

Unfortunately, tracking the virus assembly process at these more physiologically relevant locations at the top PM presents new challenges. First, the large background fluorescence due to unassembled Gag obscures the punctate signal. Second, unlike for the flat bottom PM, only a fraction of the curved top PM will be in focus when imaging with a high-NA objective. This narrow axial range both limits the number of puncta that can be imaged at one time and makes long-term experiments prone to losing track of individual puncta as they move out of the focal plane, whether due to diffusion along the curved surface or large-scale cell motions. To address these challenges, this study describes a simple method for localizing punctate objects in three-dimensional (3D) in the presence of strong, structured background. As a proof of concept, we have applied this imaging approach to the observation of HIV-1 particle assembly using a fluorescently labeled HIV-1 Gag at both the top and the bottom PM.

Because of the profile at the top surface of cells, 3D localization information is necessary, and an extended depth of field makes measurements at the top PM much more tractable. Conveniently, certain engineered point spread functions (PSFs) meet both these requirements simultaneously. We have employed the double helix PSF (DHPSF), which is one of the most common of these engineered PSFs. The DHPSF encodes axial position information in its shape consisting of two lobes that rotate as a function of the z position and significantly extend the depth of field compared with the standard PSF. However, the complex shape of the DHPSF requires specialized algorithms for both detection and localization, and the presence of structured background only makes these challenges more severe, as several recent studies have indicated (13,14).

Structured background presents an especially daunting hurdle to the detection and localization of single puncta. The quality of localization depends crucially on the signal/background ratio, and artifacts in localization can be severe (13,15). Additionally, puncta are obscured by the structured background, making identification of individual puncta more difficult. Because of the many spatial scales contributing to this type of background, it may overlap with the PSF in frequency space, increasing the likelihood of false identifications when using—for example—wavelet-based identification algorithms. The traditional approaches to circumventing this difficulty include background subtraction/removal, but this task is far from trivial with structured background (13,14).

To address both these problems simultaneously, we have employed deep neural networks (DNNs) for image analysis. DNNs have proved very successful in recognizing objects of interest in complex and varying backgrounds, and even in separating structured background from PSF images in the context of single-molecule localization microscopy (SMLM) (13). DNNs have also been shown to be effective in the regression task of localization for SMLM with a variety of PSFs due to their property as universal function approximators. We have combined these properties of DNNs in a two-step pipeline that first identifies instances of the PSF in images including complex structured background and then localizes them with high precision. This project demonstrates the possibility for DNNs to not only localize single punctate objects over relatively suppressed background, as in SMLM, but also to localize bright puncta over bright, structured, and time-dependent backgrounds without applying separate background subtraction, enabling their application in single-particle tracking in complex environments.

This image analysis workflow was then applied to monitor HIV-1 particle assembly at both the top and bottom PM of adherent cells to evaluate the performance imaging strategy. We describe the tracking of individual Gag puncta across an extended axial range in the presence of structured background. Intriguingly, we observed higher mobility and a higher likelihood of faster assembly at the top PM compared with the bottom PM, indicating that the extracellular environment may potentially affect assembly properties.

MATERIALS AND METHODS

Optical setup

Imaging experiments were performed on a custom-built inverted fluorescence microscope (Ti-2e, Nikon, Melville, NY) equipped with a high-NA (100×/1.49 NA) oil immersion objective (CFI SR HP Apo TIRF 100XC Oil, Nikon), a motorized x-y stage with a piezo z motor (MS2000, ASI, Eugene, OR), and a sCMOS camera (Prime95B, Photometrics, Tucson, AZ). A 4f system consisting of two f = 75 mm lenses (AC254–075-ML, Thorlabs, Newton, NJ) with a transmissive phase mask (DH1, Double Helix Optics, Boulder, CO) at the intermediate plane was inserted between the microscope side port and the camera as shown schematically in Fig. 1A. A broadband LED light source (UHP-M, Prizmatix, Holon, Israel) was coupled into the back focal plane of the objective and an appropriate excitation wavelength band was selected with a combination of excitation and emission filters (FF01–562/40–25 and FF02–641/75–25, Semrock, Rochester, NY) and dichroic mirror (Di03-R594-t1–25×36, Semrock).

FIGURE 1.

Optical layout of DHPSF microscope with representative images. (A) Schematic optical layout. The transmissive phase mask is placed at the intermediate plane of a 4f system at the exit port of the microscope. (B) Illustration of effect of changing defocus on the standard PSF (top) and DHPSF (bottom). Numbers above the images denote defocus in μm relative to the middle panel. (C) DHPSF image of a cell expressing HIV-1 Gag. Insets show surface plots of the image intensity data in the two red-bordered ROIs each containing a Gag punctum. Locations of the double lobes are indicated by arrows in the surface plots. Scale bar, 5 μm.

Sample preparation

U2OS cells (ATCC, Manassas, VA) were maintained in Dulbecco’s modified Eagle’s medium with 10% fetal bovine serum (Gibco, Waltham, MA) and seeded into 8-well plates with no. 1.5 glass bottoms (Cellvis, Mountain View, CA) 1–2 days before measurement. The cells were transiently transfected with HIV-1 Gag-EYFP or HIV-1 G2A Gag-EYFP using GenJet (SignaGen Laboratories, Rockville, MD) following the manufacturer’s instructions approximately 12–24 h before beginning the measurement. The transfected cells were transferred to Leibovitz’s L-15 medium (Thermo Fisher Scientific, Waltham, MA) with 10% fetal bovine serum immediately before the beginning of the measurement. The HIV-1 Gag-EYFP and HIV-1 G2A Gag-EYFP constructs have been described previously (16).

Bead z-stacks for use in the phase retrieval measurements were prepared by treating an 8-well glass-bottom plate with poly-L-lysine for 5 min before adding red fluorescent microspheres of 100 nm diameter (FluoSpheres, F8801 with excitation and emission peaks at 580 and 605, respectively, Thermo Fisher Scientific) diluted in deionized water and incubating for 20 min. The well was subsequently drained and refilled with 200 μL deionized water.

Virus-like particle (VLP) samples were purified from the growth medium of U2OS cells expressing HIV-1 Gag-EYFP 72 h after transfection. The medium was aspirated from the adhered cells, filtered with a 400 μm syringe filter, then centrifuged at 20,000 × g for 3 h before being resuspended at 20× concentration in PBS. VLPs were imaged on a coverslip that had been treated with poly-L-lysine for 5 min. The coverslip was sealed to a slide with nail polish before imaging.

Data acquisition

For retroviral Gag imaging in live cells, pre-seeded and transfected 8-well plates were mounted onto a heated stage (Lab-Tek 2000 with TempController 2000–1, PeCon, Erbach, Germany) maintained at 37°C. Gag assembly image sequences were collected on a sCMOS camera (Prime 95B, Photometrics, Tucson, AZ) with a 300 ms exposure time and 30 s off time between exposures during which the excitation light source was turned off. VLP images were collected with the same exposure and illumination conditions.

Network architecture

Our analysis pipeline has two stages (Fig. 2C). The first stage detects instances of the DHPSF in a full-size image and returns bounding box coordinates for each identified event. The object detection stage architecture is a RetinaNet (17) modified by reducing the number of channels in the input layer to take single-channel grayscale images as input. We used Resnet50 as the backbone for our RetinaNet in this work. The input image size of the images was 400 × 400 pixels, which is small enough for the GPU memory of the Nvidia GTX 1080 card while aligning with the typical dimensions of U2OS cells. The bounding box locations returned by the RetinaNet stage are used to crop images of single emitters, which are passed as a stack of 32 × 32 pixels to the second stage, which performs the precise localizations and intensity estimates. The second stage consists of two custom light-weight convolutional neural networks (CNNs) which return a list of coordinates and intensities for each event. The DNN was implemented with the PyTorch library. Details of the network architecture, hyperparameters, and training conditions are presented in Fig. S4 and Tables S1–S4. Code for the network is available at https://github.com/fluorescencemicroscopy/CBNet.

FIGURE 2.

Overview of DNN analysis pipeline. (A) DHPSF training images are generated from the pupil function of our microscope using scalar diffraction theory. A uniform background is added, and noise is applied to model the shot noise and readout noise of real images. We refer to the network trained on these DHPSF-only images as UBNet. (B) Images designed to train the network to handle HIV-1 Gag cell data are created by adding CB noise images with sharp edges (which already includes shot noise but not readout noise as described in materials and methods) to DHPSF images as described in (A). We refer to the network trained on these images as CBNet. (C) The analysis pipeline takes full-size images from the microscope (left panel) and recognizes individual DHPSF objects (middle panel). Regions recognized in the first step are cropped and passed to the second stage, which returns x, y, z, and intensity estimates for each region identified in the first stage.

Phase retrieval

The pupil function of the microscope including the phase mask was estimated experimentally using VIPR, a recently described software for phase retrieval based on vectorial diffraction theory (18). Fluorescent microspheres (FluoSpheres, Thermo Fisher Scientific) were immobilized on a glass coverslip as described above. Images were recorded by the camera at the exit port of the 4f system (see Fig. 1A) at a series of different heights. The height displacements of the coverslip were achieved using the piezo z-drive on the stage (ASI). Images and their corresponding heights provided by the piezo controller were used by the VIPR software to estimate the pupil function of the microscope, which includes the phase mask as well as any other optical aberrations that may be present.

Training data

DHPSF images for training were created computationally using the pupil function estimated by phase retrieval. Using a previously described imaging model based on scalar diffraction theory (19), images of variable numbers of point-like objects were computed at random positions and intensities. The positions were selected from a uniform distribution, excluding the 20 pixels at the edges of the image to avoid truncating PSF images. Each emitter was assigned a photon count chosen randomly from a uniform distribution between 2850 and 50,000 photons. A uniform background of between 50 and 400 counts was added to the image. Next, Poisson noise was applied to the image to model shot noise, and Gaussian noise was added to model the sCMOS readout noise, with its mean representing the camera offset. While the Gaussian model is only an approximation for the readout noise, it describes the distribution well with the exception of a very small number of outliers (Fig. S1). Because the amplitude of the readout noise is small compared with the shot noise contribution under the acquisition conditions here, these deviations from normality are expected to have a minor effect on network performance. An example 400 × 400 pixel training image containing DHPSF point objects generated by this procedure is shown in Fig. 2A.

Structured background images were computed separately using a procedure based on the simplex noise algorithm (20). The simplex noise was calculated for two dimensions over seven octaves with lacunarity set to 1.9 and persistence set to 0.1 (Fig. S2). To imitate the effect of cell edges in the computational structured background, a sharp boundary was introduced to the simulated complex background (CB) images. This procedure is described in Fig. S2. Briefly, curvature is incorporated into the starting simplex noise image, leaving the edges of the image with lower values on average. Pixel values that fall below a threshold are made flat and separated from the noise-filled region to create a sharp boundary. Poisson noise is applied in the last step to create the final, cell-like CB image. Examples of generated complex structured background images are shown in Figs. 2 B and S2. Parameters used to generate the training data are listed in Table S1.

BG complexity

Following the definition of structural complexity used in a recent study (21), we consider the mean gradient of an image as a simple metric of the background (BG) complexity of an image. We calculated our BG score in a 32 × 32 pixel region of structured background images (without DHPSFs) using the Sobel operator to approximate the gradient. We first subtracted the camera offset, then took the mean value over the whole 32 × 32 pixel region of the magnitude of the Sobel operator applied to that image as the BG score. Example regions of structured background images are shown in Fig. S8 along with the corresponding BG scores.

CRLB calculation

The calculation of the Cramér-Rao lower bound (CRLB) for images has been described in detail elsewhere (22,23). We followed the calculation of (24), with readout noise of the sCMOS camera modeled as Gaussian. Because some fraction of the intensity from the DHPSF implemented on our microscope falls outside the 32 × 32 pixel region centered at the punctum location that is considered by the localization DNN, especially toward the edge of the z range (Fig. S6), these counts do not contribute to the localization. To account for this effect, we adjusted the total counts contributing to the CRLB calculation to only include those in the central 32 × 32 pixel region. The fraction of counts falling inside this region is shown as a function of the z position in Fig. S6.

Data analysis

Image stacks collected from the camera on the microscope were sent to CBNet for processing, yielding a list of the detected puncta’s frame numbers, positions, and intensities. Individual puncta were tracked through time by linking detections across frames using a combination of automatic tracking using the LAP tracker function provided by TrackMate (25) and manual supervision. Puncta to be tracked were selected from regions of low density where individual puncta could be unambiguously linked between frames. Tracks were manually reviewed by comparing the raw image data to putative tracks and manually updated as needed. Only tracks with a temporal intensity profile that showed initial growth followed by a plateau were fitted to a saturating exponential function to estimate assembly kinetics. Occasionally tracks were observed with a saturating exponential signature whose plateau was followed by additional changes in the intensity or whose fast growth phase followed an initial period of unchanging intensity. In these cases, only the part of the track well described by the saturating exponential function was fitted.

RESULTS

We implemented a 3D imaging system using an engineered PSF by placing a transmissive phase mask encoding the DHPSF at the Fourier plane of a 4f system at the exit of the microscope (Fig. 1A). The DHPSF phase mask creates an aberrated image of point-like emitters that depends on the axial position. Specifically, a point-like object viewed under the DHPSF microscope appears as two lobes that rotate about their center as the axial position of the emitter changes (Fig. 1B). The rotation angle of the DHPSF lobes thus encodes the axial position of the point object (26,27). Localizing point-like emitters using the DHPSF presents extra challenges compared with the standard PSF, especially in the presence of complex structured backgrounds such as those that exist in cells expressing human retroviral Gag (13,28). Fig. 1C shows an example of a cell that has been transiently transfected with fluorescently labeled HIV-1 Gag. A strong, nonuniform background that exhibits local structure exists throughout the cell. This background is generated from unassembled, fluorescently labeled Gag protein (2,29). Punctate HIV-1 Gag represents multimerized Gag at the PM, a few examples of which are indicated by red boxes in Fig. 1C. The double lobes of the Gag puncta are faint and obscured by the nonpunctate background, whose complicated and varied shape is shown in the surface plots in Fig. 1C.

We initially attempted to localize the Gag puncta using Gaussian least-squares fitting to the two lobes of the DHPSF. For this task we chose Easy-DHPSF (30), which was designed for localization of DHPSF PALM/STORM data without structured background. Easy-DHPSF identifies individual candidate point-like objects by a wavelet-based template-matching algorithm, then localizes the objects by least-squares fitting. We found that the wavelet-based template matching was not well suited to localizing puncta in this experimental system because of the structured background. Many false positives were identified in areas containing structured background, often at areas with sharp boundaries near the edge of the cell and around the nucleus (Fig. S3). In addition, many DHPSF events visible by eye in the images were not identified.

To overcome this challenge, we turned to deep learning as an alternative approach to localize DHPSF data in the presence of complex structured background. We designed a DNN that provides an end-to-end pipeline for 3D localization microscopy analysis to identify and localize puncta and provide intensity estimates in the presence of strong structured background as illustrated in Fig. 1C. DNNs have shown promise in object detection and superresolution microscopy (19,31–33), but require large data sets for training. Furthermore, ground truth information is not available for experimental data sets. Therefore, we used a computational approach to generate training data, creating images of individual DHPSFs calculated using scalar diffraction theory (19), with the microscope pupil function estimated from phase retrieval (18). These calculated DHPSF images over uniform background were used to train our DNN pipeline (Fig. 2A), which we refer to as Uniform Background Net (UBNet). While we expected UBNet to perform well on images without complex structured, nonuniform background, we anticipated it to be insufficient for images containing DHPSFs in the presence of complex structured backgrounds such as shown in Fig. 1C. Therefore, we created a separate data set with DHPSF images embedded in computationally generated CBs designed to mimic the fluorescent background typical in cells expressing fluorescent HIV-1 Gag. We developed a custom algorithm to create the cell-like background images based on the simplex noise algorithm (20). This procedure, which we refer to as cell-like CB noise, adds sharp edges to mimic the edges of the cell and nucleus and a gentle slope away from near the center of the image to mimic realistic cell features. The CB algorithm is described in the materials and methods and in Fig. S2. We expected that training our pipeline on DHPSF images in the presence of realistic cell-like background would allow it to learn to discriminate between the PSF of interest and the cellular background. Therefore, we also trained our DNN pipeline on DHPSF images containing CB noise (Fig. 2B), which we refer to as Complex Background Net (CBNet).

Our neural network consists of a two-stage pipeline that takes full images without preprocessing from the microscope as input (Fig. 2C). The first stage of the pipeline is an object detection stage, which identifies individual DHPSF objects in the full image. This first stage uses a custom grayscale implementation of the RetinaNet architecture, which is a simple single-stage dense object detection network that produces fast and accurate results (17). The architecture of the CNN is described in Fig. S4. The image regions identified by the first stage of the network are passed as 32 × 32 pixel subimages to the second stage, which provides estimates of the x, y, and z coordinates and intensities of individual DHPSF instances. The location and intensity networks consist of custom designed lightweight CNNs that take the 32 × 32 pixel output of the first stage as inputs and return coordinates and intensity estimates as their outputs (Fig. S4).

We validated the performance of our trained UBNet and CBNet in several ways. First, we applied them to simulated DHPSF images without structured background (Fig. 2A). Specifically, in this simplest case images were generated with a single DHPSF object of constant intensity and with a uniform and constant noise level. Under these conditions, the minimum uncertainty for an unbiased estimator can be calculated, called the CRLB. Because the shape of the DHPSF varies as a function of z, we compared the standard deviations of the parameter estimates returned by the networks to the CRLB at a series of z positions. We generated images with DHPSFs of an intensity of 15,000 counts over a uniform background of 100 counts, with readout noise of the camera modeled to be normally distributed with a standard deviation set to match the parameters for our sCMOS camera. We first examined the results of applying UBNet to this data set similar to its training data set, shown in Fig. 3A (empty squares). The standard deviations of UBNet’s estimates closely matched the CRLB for all three spatial coordinates and intensity across the axial range of the PSF, demonstrating that UBNet localizes the DHPSF objects with the best precision allowed by information theory. Similarly, we observed that UBNet approached the CRLB across a range of different test conditions (Fig. S5). These results verified that our network architecture was capable of ideal performance under simple conditions. Next, we also examined CBNet results on the same data set (Fig. 3A, solid squares). The observed standard deviations in this case were close to the CRLB and follow its trend as a function of z, with a small gap separating them (of ∼<10 nm for the spatial coordinates and ∼50 photons for the intensity). This result showed that the network provides excellent estimates, and only a relatively minor loss in precision compared with the ideal case.

FIGURE 3.

Precision of analysis pipeline on test data sets with and without complex background noise. Standard deviations of x, y, z, and intensity estimates (A) of DHPSF images with 15,000 counts and a background of 100 counts and no complex background at a series of z positions are plotted alongside the CRLB (green line). The results of CBNet are plotted as solid squares, while the result of UBNet are shown as empty squares. (B) Precision of CBNet’s estimates for DHPSF images with 15,000 counts over a range of backgrounds generated by the CB algorithm. The CRLB from (A) is plotted as a green line for comparison.

Confident that the network design is sound for data lacking structured CB noise, we next evaluated the performance of the network on computationally generated images including structured CB noise (Fig. 2B). Following a similar approach to the previous test, we generated test images with DHPSFs with an intensity of 15,000 counts at a series of z values, this time including CB noise to model structured background. Unfortunately, with the addition of structured background, we could no longer calculate the CRLB to have an absolute comparison for the network performance. However, the results of this test give an indication of the network under a variety of realistic conditions. Fig. 3B shows the standard deviations of CBNet’s estimates of the spatial coordinates (x, y, and z) and intensities for this test data set, which included CB noise with a wide range of amplitudes (see materials and methods and Fig. S2). For comparison, the CRLB calculated in the absence of structured background is also plotted, although it cannot be interpreted as an attainable precision for these data. Unsurprisingly, the localization precision is reduced compared with the data set lacking structured background, to ∼20–25 nm in x-y and ∼30–40 nm in z across much of the z range. The standard deviations of the estimates are lowest in the middle of the z range and increase toward the ends of the range, following the shape, at an offset, of the CRLB calculated in the absence of CB noise.

Another important consideration in evaluating the network performance is its success in identifying DHPSF objects, especially in the presence of CBs. In the absence of CB noise (data set of Fig. 3A), CBNet identified virtually all DHPSF objects correctly. With the structured background (data set of Fig. 3B), a small fraction of the events was not identified, with the highest proportion of false negatives occurring near the edges of the z range. The lowest fraction of missed events had z positions around the focal plane, with around 0.1% of events missed across the central range (∼1 μm), while the highest fraction of false negatives was around 10% at the highest z value (Fig. S7). These results are consistent with the larger footprint of the DHPSF at extreme z positions, with fewer photons concentrated in the lobes than at positions close to the focal plane.

Next, we investigated the effect of the structured background on the identification and precision of estimates from UBNet on individual DHPSFs. The structural complexity measures the intricacy of an image and reflects the local variability in pixel values (21), and is expected to affect the quality of localization and estimation (13). We used the mean value of a Sobel filter (see materials and methods) as a simple approximation of the mean gradient (21) measure of the structural complexity. We refer to this quantity as the BG score. Images that are steeper and more intricate have higher BG scores, while those that are flatter, such as in Fig. 2A (middle), have low BG scores. Examples of various images with their BG scores are shown in Fig. S8. We examined the CBNet results on this same data set as a function of the BG score of the images (Fig. 4). We arbitrarily grouped BG scores into three regimes: low, with relatively featureless backgrounds, medium with some features obvious, and high with steep features. Examples of each regime are shown in the left column of Fig. 4. The fraction of events identified in this test data set dropped with increasing BG score, from near 100% in the low BG regime to ∼94% over high backgrounds (Fig. 4). The standard deviations of location and intensity estimates increased with increasing BG score, from values comparable with the precisions observed for CBNet in the absence of CB noise (Fig. 3A) at the lowest BG score to substantially higher values at the highest BG scores (Fig. 4).

FIGURE 4.

Effect of complex structured background on position and intensity estimates. Left: examples of backgrounds with low, medium, and high BG scores (labeled on images), respectively. Top right: fraction of events identified as a function of BG score. Bottom right: standard deviations of location and intensity estimates as a function of BG score.

The previous tests of CBNet in the presence of CB backgrounds used a fixed intensity at a series of fixed positions. To evaluate the performance of the network under more general conditions, we generated another test data set with relaxed constraints and compared network estimates to the ground truth values. Instead of fixing z positions and intensities, we placed a random number of PSFs between 0 and 15 per image, at random positions and with random intensities drawn from across the range of training conditions over CB backgrounds. We then compared the estimates returned by our analysis pipeline with the ground truth values, considering any estimate within 250 nm in x, y, and z and a true positive identification. The estimated coordinates and intensities for these events are plotted against the ground truth values in Fig. 5. CBNet identified 6300 of the 6928 DHPSFs (91%) to within 250 nm in all coordinates, while failing to identify 628 (9%) and returning 239 (3.4%) false positives (events more than 250 nm in any coordinate from a ground truth position). Of the true positive identifications, the standard deviations of the differences between the estimates and ground truth coordinate were 23, 21, and 41 nm in x, y, and z. The corresponding standard deviation for the intensity was 1494 counts.

FIGURE 5.

Network performance in the presence of structured background over a range of conditions. (A–C) Scatterplots of the position estimate returned by CBNet plotted against the ground truth (GT) value for x, y, and z, respectively. The red line shows the GT value. Insets show histograms of the estimate minus the GT value. (D) The corresponding plot for intensity estimates.

These results verified that our network design performs well with data that include computationally generated CB backgrounds. However, we wanted to validate its performance in the presence of structured CBs from real cell images of fluorescent HIV-1 Gag. Because ground truth information is not available for experimental data, we used a hybrid approach of generating DHPSF point objects computationally and adding them to experimentally acquired images of fluorescent Gag backgrounds lacking puncta. We collected a series of images of a membrane binding-deficient mutant of HIV-1 Gag (G2A Gag mutant) to provide examples of real structured background without puncta (16,34). Then, we added a series of DHPSF point objects at fixed positions with intensity increasing following a saturating exponential curve as a function of time to mimic the behavior seen with Gag puncta (2–4). Fig. 6A shows a few examples of the augmented data prepared with this hybrid approach over low to medium BG score regions ranging from 60 to 98, while Fig. 6B plots the intensity estimates derived from CBNet as points along with the ground truth intensity (red lines) for the images in Fig. 6A. The resulting estimates of the intensity curves (Fig. 6B) matched well to the ground truth curves, with a standard deviation of the fractional intensity error of 7.9%. Additionally, the position estimates gave ∼25 nm precision for the x and y coordinates and ∼60 nm for the z coordinate.

FIGURE 6.

Network performance on augmented data. (A) Three augmented images with computationally generated DHPSFs combined with experimental images of a cell expressing HIV-1 Gag G2A. The locations of four synthetic puncta are highlighted by the red boxes. (B) CBNet estimates of intensity for the puncta highlighted in (A) (blue squares) and the ground truth intensities (red lines).

The results of these performance tests gave us confidence to apply CBNet to analyze the biogenesis of Gag puncta. To observe Gag puncta biogenesis in live cells we transiently transfected HIV-1 Gag-EYFP into U2OS cells and placed the cells on a heated stage on the microscope. We then collected time-lapse images of the cells, with a 30 s gap between images to facilitate long-term measurements typically lasting several hours. Expression of fluorescence in different cells started at different times, with newly fluorescent cells becoming visible for many hours after the appearance of the first. Cells early in expression had a diffuse cytoplasmic fluorescence signal as has been described before (2). The cytoplasmic fluorescence background signal grew stronger over time, and we often observed the appearance of many puncta at the PM in quick succession and at a similar cytoplasmic intensity across cells. A single frame of a movie from a cell expressing HIV-1 Gag-EYFP observed with the DHPSF is shown in Fig. 7A with the focus near the top PM. Applying our network to each frame of this movie, we identified, localized, and extracted intensities for individual puncta in the live cell. A few frames at specific time points from a subregion of Fig. 7A are shown in Fig. 7B, together with puncta identified by CBNet. These identified events were linked together across different frames to construct trajectories. The intensity profiles from the two puncta identified in Fig. 7B are shown as examples in Fig. 7C. Following previous reports (3), we fit the intensity profiles to a saturating exponential function of the form A(1 − exp(− (t − t₀) /t_c)) to recover assembly times of 14 and 17 min. The spatial trajectories of the two puncta from Fig. 7B show directed motion (Fig. 7D), which we often observed for puncta at the top membrane. The Gag puncta shown in Fig. 7D were first detected near the periphery of the cell and moved toward the nucleus over time, noticeably climbing in z as they reach the edge of the nucleus. Notably, many puncta remained associated with the cell for a long time after the intensity had saturated (Fig. 7C), as has been previously reported (2,3).

FIGURE 7.

Fluorescently labeled HIV-1 Gag assembly in a live U2OS cell. (A) An example frame from a movie of a cell expressing HIV-1 Gag-EYFP. (B) Enlarged view of the red boxed region from (A) at a few selected time points. Two puncta are highlighted in orange circles and purple pentagons. The last image at t = 130 min identified three additional puncta (green squares). (C) Intensity profiles of the two puncta highlighted in (B). (D) Location trajectories of the puncta highlighted in (B).

For comparison, we also measured Gag puncta formation on the bottom membrane adjacent to the coverglass using the same DHPSF setup. The spatial trajectories of the puncta on the bottom membrane showed a striking difference from those at the top PM. While trajectories at the top membrane almost universally displayed directed motion, such as those shown in Fig. 7D, often with a displacement of several microns or more during the course of observation, those at the bottom PM were mostly stationary by comparison. Fig. 8A and B show a few examples of typical x-y trajectories of puncta on the top and bottom PM, respectively. Further example trajectories are shown in Figs. S9 and S10, along with their intensity profiles in Figs. S11 and S12. The puncta at the bottom, in addition to having smaller total displacements than those at the top, in general did not show any signs of preferred direction of motion, instead giving trajectories that appear consistent with random, Brownian motion. These impressions are confirmed by the mean-squared displacement (MSD) plots for these Gag puncta (Fig. 8C), which clearly show the larger displacements characteristic of the top PM puncta (blue lines) compared with the bottom PM Gag puncta (orange lines). In addition, the MSD curves for the top PM Gag puncta increase faster than linearly, indicating active, directed motion. Those at the bottom, by contrast, increase approximately linearly, consistent with random motion. Furthermore, by fitting the intensity profiles of the two groups of Gag puncta as described above (46 tracks on the top PM from 16 cells on 9 different days and 50 tracks on the bottom PM from 7 different cells on 3 different days), we characterized the assembly kinetics in terms of the characteristic time t_c of the fits (Fig. 8D). Interestingly, the assembly times of Gag puncta at the top PM were skewed to faster t_c than the puncta at the bottom. The distributions of Gag puncta formation times at the two PMs were significantly different, with a p value in the two-sided Kolmogorov-Smirnov test of ∼7 × 10^—6. The cumulative distribution functions for the two distributions are shown in Fig. S13. The saturating intensities of the Gag puncta in the cell were compared with the intensities of purified VLPs observed under the same imaging conditions (Fig. S14). While the observed distributions were wide for both the saturating values of individual puncta in cells and the VLPs, the distributions of both were similar to each other as was observed in a previous study (3). This observation suggests that the amount of Gag in the puncta observed in cells is consistent with that of released VLPs.

FIGURE 8.

Differences observed between top and bottom PM. (A) Examples of x-y trajectories recorded at the top PM. (B) Examples of x-y trajectories recorded at the bottom PM at the same scale as (A). Scale bar, 5 μm. (C) MSD plot of the trajectories in (A) and (B), showing the difference in mobility of puncta from the top membrane (blue) and the bottom membrane (orange). Inset, an enlarged view of the MSD plots of the puncta on the bottom PM. The MSD plots of puncta at the top membrane are superlinear, while those at the bottom PM are linear at small t and in some cases plateaued, suggesting confinement. (D) Histogram of assembly times observed at the top and bottom PM. The distribution of assembly times at the top PM is skewed to fast assembly, with a significantly higher population of fast assembly times compared with the bottom PM.

DISCUSSION

Detection and localization of the DHPSF and other engineered PSFs with complex shapes is challenging compared with the standard microscope PSF. While various methods have been described to circumvent the lack of analytical expressions for these engineered PSFs, including phase retrieval-based maximum likelihood estimation and spline-based modeling of the PSF (30,35,36), these approaches are typically computationally expensive and require time-consuming calibration procedures and manual user input to set thresholds. Deep learning has proved to be a promising approach to overcome these shortcomings in recent years, demonstrating success in localizing single emitters in the context of SMLM (19,31–33,37,38).

In the presence of structured background, however, the analysis challenge is heightened because the PSF model cannot account for the structure in the background. Instead, the local background is often approximated as constant for convenience. However, such an approximation can strongly influence the resulting parameter estimates (13,15). Unlike uniform background, structured backgrounds are nontrivial to subtract because the multiple spatial scales of the structure must be disentangled from the spatial structure of the PSF itself. A recent study described a deep learning-based method to separate structured background from the PSF, but it requires regions of interest containing a single punctum to be first identified in a full image (13). However, identifying punctate PSF objects in an image containing structured backgrounds is often far from straightforward, as illustrated in Fig. 1C, creating a significant challenge for the efficient analysis of the data. A recent study, recognizing this issue, described a method that addresses this problem for the standard microscope PSF (14), but since the approach relies on the radial symmetry of that PSF, it cannot be applied to the existing engineered PSFs. Thus, to the best of our knowledge, no suitable tool for this task has been introduced.

To remedy this situation, we designed a DNN pipeline that first identifies punctate objects within the image containing structured background and then determines the 3D location and intensity of each of the individual DHPSF objects. Our pipeline proved to be capable of optimal localization precision in the ideal case of uniform background (Fig. 3A) if trained as such (UBNet) and only experienced a moderate loss in precision when trained on complex structured background (CBNet). Testing our network in the presence of computationally generated structured background revealed a further loss in precision by roughly a factor of 3 (Fig. 3B). To better quantify this loss, we introduced the CB score to judge the severity of the CB surrounding a DHPSF point image, which is similar to other measures of complex structured background described in the literature (21). The uncertainty in determining position and intensity of DHPSF point objects scales approximately linearly with the CB score of the image background (Fig. 4), while the detection efficiency appeared largely unaffected by the background at low BG score before dropping to ∼93% at the highest values we tested (Fig. 4). The number of false positives in the test set was under 1% of the number of events correctly identified, resulting in a Jaccard index for the whole data set of 0.97.

To bridge the gap between synthetic and experimental data, the trained network was applied to synthetic fluorescent puncta superimposed over real cytoplasmic background to mimic the assembly of puncta in cells expressing HIV-1 Gag-EYFP (Fig. 6). The close resemblance of the DNN output to the ground truth reassured us of the soundness of the DNN architecture and encouraged us to apply CBNet to localize and estimate intensities of individual HIV-1 Gag puncta in live cells in the presence of a large CB. The pipeline succeeded in localizing puncta in live cells. By linking the recovered locations and intensities from each frame for individual puncta, the temporal evolution of the assembly process could be evaluated on the single punctum level. The growth in the intensity represents the increase in the copy number of HIV-1 Gag-EYFP of a particular punctum (Fig. 7C), which provides information about the kinetics of the process, while the location of the punctum yields the spatial trajectory of the HIV-1 Gag punctum biogenesis at the PM. Similar to previous reports that monitored HIV-1 Gag punctum biogenesis, we observed that the intensity of Gag puncta formation initially increased quickly and then plateaued (2–4). There was a high degree of heterogeneity in the Gag punctum formation times, as has been reported previously (3,4).

While the distribution of times we observed overlaps with the range of reported values (Jouvenet et al. reported 5–6 min assembly with an average of 8.5 min (2), Ivanchenko et al. reported 8–9 min assembly with a wide distribution going up to ∼40 min (3), and Ku et al. also reported a broad distribution ranging from ∼10 to 60 min (4)), our measurements also included a range of somewhat longer assembly times. In contrast to previous studies, we transfected 100% HIV-1 Gag-EYFP rather than a mixture of HIV-1 Gag-EYFP and unlabeled HIV-1 Gag, which an earlier TEM study found resulted in defects in the morphology of the immature Gag lattice (39). We also used a different cell line, U2OS, from previous reports and measured over very long times. These factors may help to explain the differences between Gag punctum assembly times we observed and previous reports. Additionally, previous measurements used a variety of methods to estimate assembly times, complicating direct comparison. Our study found that the assembly kinetics observed at the top membrane are skewed toward faster assembly than at the bottom membrane. Like-wise, a recent study using scanning ion conductance microscopy reported significantly faster assembly kinetics at the top membrane (∼30 s assembly time) (40), although the absolute times reported were faster than we observed here. While these differences in assembly kinetics could reflect variations in the cytoskeletal stiffness at adherent and non-adherent surfaces (8,40), the fundamental origins of these observed differences are currently not understood. In addition, we observed that puncta on the top PM moved a substantial distance over the course of the experiments in a directed fashion, similar to what has been observed previously using atomic force microscopy (10).

Despite the advantages of DHPSF imaging coupled with deep learning analysis compared with TIRF, which by its nature is inherently limited to the glass-cell interface, its use involves some tradeoffs. Without the background suppression inherent to TIRF microscopy, our approach suffers from increased background, meaning that TIRF detects Gag puncta biogenesis earlier than the DHPSF method. However, the main drawback of DHPSF imaging is the larger footprint of the PSF, which both reduces the density of objects that may be imaged without overlapping PSFs by roughly an order of magnitude and reduces the signal/noise ratio by spreading the signal photons over a larger area compared with a standard microscope PSF. While our analysis pipeline performed well overall, it has limitations at the highest Gag punctum densities we tested. For data sets where PSF overlap is expected to be significant, other second-stage architectures may perform better than our lightweight CNN (19). Incorporating such an alternative localization network into our analysis pipeline would be straightforward.

Although this study focused on the development of combining DHPSF imaging with a deep learning analysis pipeline to enable localization of punctate objects in three dimensions even in the presence of structured background, we observed intriguing hints that differences could exist in both the mobility and kinetics of Gag punctum assembly at the top and bottom PM. The behaviors of individual puncta were highly heterogeneous, indicating that the assembly site location may affect these properties, but is not the only factor. Our results indicate that the local extracellular environment may affect Gag puncta biogenesis, but further investigation will be necessary to determine how general the effects are and to elucidate the mechanisms. The tools described here provide the foundation for further studies to probe these questions by extending into more physiologically relevant environments. This would include measurements in the natural targets of HIV-1 infection—i.e., primary monocytic or T cells—as well as the use of infectious HIV-1 molecular clones expressing a labeled Gag protein.

In summary, coupling DHPSF imaging with a properly trained DNN is well suited for broad studies of punctate, viral, or cellular assemblies in the presence of CB signals. The method is technically easy to implement, requiring only epifluorescence illumination and the addition of a commercially available DHPSF phase mask in a 4f configuration before the camera. The analysis pipeline itself can be easily retrained for use with different engineered PSFs. Once the CBNet has been trained, no further manual calibration steps are required. Running the inference step is fast, with each frame taking ∼60 ms for our 400 × 400 pixel images with only a weak dependence on the punctum density. The size of our image was dictated by the limited memory of our GPU card, but larger images could be accommodated by newer GPUs. Because of the extended depth of field of the DHPSF, we do not need to take z-stacks to cover the range imaged, allowing fast, real-time imaging over an extended axial range. While we were motivated to develop CBNet to study HIV-1 Gag punctum biogenesis, our analysis pipeline could be easily applied to other systems. The crucial considerations required to apply this method are punctate structure and low enough density to avoid overlapping PSFs. The signal and background levels also contribute to the likelihood of detection of individual structures, with increasing background complexity reducing the fraction of detected objects (Fig. 4). For example, a similar experiment could probe the dynamics of viral attachment and virus entry by tracking localized fluorescently labeled viral proteins that self-assemble over time. Other punctate systems would also be candidates for this analysis, including MS2-labeled RNAs, which would be useful to study HIV-1 RNA genome recognition and viral RNA packaging, vesicles, or chromosomal loci. This approach could also be useful in cellular studies of endosomes, exosomes, or clustered membrane proteins. Thus, our approach is fast, convenient, and flexible, and we expect that it will complement existing techniques in providing robust 3D localization even in the presence of structured background.

Supplementary Material

Supporting material can be found online at https://doi.org/10.1016/j.bpj.2025.08.028.

SIGNIFICANCE.

Engineered point spread functions allow single-shot 3D localization of punctate objects over an extended axial range but require the illumination of a large volume. This illumination frequently results in complex structured backgrounds that confound the accurate localization of fluorescent puncta within biological samples. Computational removal of structured background is difficult because of the wide range of spatial scales present. To address these challenges, we combined the double helix point spread function with deep learning analysis to enable localization microscopy in the presence of structured backgrounds. Using this analysis pipeline, assembly kinetics of HIV-1 Gag puncta were measured at both top and bottom plasma membranes.