Machine learning for cross-scale microscopy of viruses

Summary

Despite advances in virological sciences and antiviral research, viruses continue to emerge, circulate, and threaten public health. We still lack a comprehensive understanding of how cells and individuals remain susceptible to infectious agents. This deficiency is in part due to the complexity of viruses, including the cell states controlling virus-host interactions. Microscopy samples distinct cellular infection stages in a multi-parametric, time-resolved manner at molecular resolution and is increasingly enhanced by machine learning and deep learning. Here we discuss how state-of-the-art artificial intelligence (AI) augments light and electron microscopy and advances virological research of cells. We describe current procedures for image denoising, object segmentation, tracking, classification, and super-resolution and showcase examples of how AI has improved the acquisition and analyses of microscopy data. The power of AI-enhanced microscopy will continue to help unravel virus infection mechanisms, develop antiviral agents, and improve viral vectors.

Graphical abstract

Petkidis et al. discuss the use of artificial intelligence (AI)-guided computational microscopy for virological sciences. AI enhances image acquisition and analyses, including object detection, tracking, and quantification. It provides insights into the underlying cell state characteristic of a particular infection phenotype, such as lytic, nonlytic, persistent, or abortive infections.

Introduction

Viruses affect all forms of life. The sheer number of virions exceeds the number of cells and organisms by orders of magnitude. The agents have caused disease in humans and animals for thousands of years and continue to emerge unpredictably. Unfortunately, we still lack a deep enough understanding of viruses to effectively treat or prevent their diseases. This is in part due to the intricate nature of virus-host interactions, which essentially involve all cellular processes. It is further complicated by the difficulty to mechanistically unravel the manifold functions of viral proteins exhibiting high binding promiscuity to host and viral factors, as shown by the adenovirus (AdV) immediate-early protein E1A.¹ Consequently, the susceptibility of cells and individuals to infection and disease is complex, as demonstrated in the SARS-CoV-2 pandemic, where hundreds of millions of individuals have been infected, yet only a fraction of them developed severe symptoms of disease, and many remained virtually asymptomatic.² The mechanisms underlying such virus infection variability include genetic factors, such as inborn errors of immunity,³ non-genetic factors, for example infection history, the metabolic state, epigenetic modulation of the viral and host genomes, protein, RNA, and lipid abundance, as well as the well-known variability of infection susceptibility of particular cell types.^4,5 All these parameters define cell states and susceptibility or resistance to infection.

The development of imaging technology has been crucial to illuminate virus infections.^6,7,8 Enormous progress in optics and photonics has led to the development of microscopes capable of imaging at a multitude of spatial and temporal scales. Recently, new advances in super-resolution microscopy (SRM) have enabled particle resolution below the diffraction limit.^9,10,11 For example, improved fluorescent probes with enhanced brightness, photostability, and spectral range greatly facilitate single virus tracking in live cells.^6,12,13,14 This progress in optics and chemistry is accompanied by increasingly sophisticated computer vision algorithms and image analysis procedures, enabling object segmentation (Figure 1A), tracking (Figure 1B), denoising (Figure 1C), and annotation of phenotypes (Figures 1D and 1E).¹⁵ Here, we discuss how AI-enhanced light and electron microscopy help to elucidate processes and cell states in viral infections.

Figure 1.

Applications of machine learning (ML) in viral image analysis

(A) Nuclear segmentation of the Hoechst 33342 DNA signal from human lung epithelial A549 cells infected with AdV using StarDist.¹⁶ The segmented image shows the overlay of the segmentation mask with the raw image. Colors indicate object instance. The white arrow points to an infected nucleus. Scale bar 20 μm. Of note, conventional methods, e.g., Otsu-thresholding or entropy-based methods, are typically unsuccessful in yielding reasonable segmentation masks for all nuclei.

(B) Tracking of fluorescently labeled AdV particles containing a capsid tag using TrackMate.¹⁷ Circles in the output show objects detected by differences of Gaussians (DoG), and lines show trajectories obtained from the linear assignment problem (LAP) tracker. Scale bars at the top and bottom indicate 5 and 1 μm, respectively. Movie courtesy of Dr. Michael Bauer.¹⁸

(C) Image denoising using Noise2Void2.¹⁹ The procedure identifies incoming viral DNA genomes in A549 cells infected with ethynyl-deoxy-cytidine (EdC)-tagged AdV-C5 and stained by click chemistry as described.²⁰ Images show maximum intensity projections of confocal stacks of an EdC staining (image courtesy of Alfonso Gómez-González). The white arrow indicates a virus particle, and the yellow box is a zoomed-in view. Scale bar, 5 μm.

(D) Functional classification of AdV-inoculated A549 cells into infected and uninfected cells based on ViResNet and the nuclear signal of the Hoechst dye.²¹ Scale bar, 10 μm.

(E) An example for data exploration using k-nearest-neighbor (k-NN) clustering.

The power of single-cell imaging of virus infections

Cell-resolved imaging gives insight into the molecular mechanisms of infection and the variability of infection phenotypes. Imaging-based methods for the study of cell infection states comprise detection of DNA, RNA, and protein species with high molecular specificity, high coverage, and ideally single-molecule resolution.⁵ For instance, copper(I)-catalyzed azide-alkyne cycloaddition (click) enables the selective visualization of incoming alkyne-tagged single viral genomes.²⁰ Likewise, spatial detection of viral and cellular transcripts has become possible by a toolbox of methods suitable to analyze features of tissues in toto. For instance, Slide-seq allows transcript detection in tissue sections with near-cellular resolution.^22,23 Meanwhile, subcellular transcription patterns can be determined by single-molecule RNA-fluorescence in situ hybridization techniques, including multiplexed error-robust FISH and seqFISH+.^24,25 Protein abundance and localization can be revealed by highly multiplexed immunostainings, imaging mass cytometry, or spatial proteomics.^26,27,28 Combined with computational image analyses, these approaches have already given important insights into the genome-to-genome variability of AdV gene expression²⁹ or the heterogeneity of influenza virus progeny in particle assembly.³⁰

Deep learning and computer vision in fluorescence microscopy

Fluorescence microscopy has been widely used to analyze biological processes, including virus infections in live or fixed cells and tissues. In recent years, machine learning (ML) algorithms have been applied to automatically extract structured knowledge from microscopy images by using supervised and unsupervised learning. In supervised learning, the algorithm detects patterns from expert-annotated datasets during training, useful for the correct classification of unseen data (Figure 1D). If no expert annotation is available, unsupervised learning groups similar images together using clustering algorithms (Figure 1E). Deep learning (DL) models, notably artificial neural networks (NNs), represent data at numerous levels of abstraction³¹ and significantly broaden the spectrum of biological problems applicable to computer vision.³² For virological research, these developments are important considering the vast heterogeneity, variability, and dynamics of infection phenotypes.⁵

In the following, we discuss how DL has advanced the interpretation of virus microscopy data and facilitates the detection and phenotypic characterization of microscopic cell patterns. We discuss how DL procedures affect all stages in microscopy, including image acquisition, analysis, and interpretation, and we highlight useful resources (Figure 2). Technical information about some of the most popular NNs in image analysis is presented (Table 1) and showcases applications of NNs to studies of the viral life cycle (Table 2). Finally, we outline future avenues for AI in the context of viruses and their infections.

Figure 2.

Applications of deep learning in microscopy

DL procedures affect all stages of image acquisition and analysis. Procedures and frameworks include event-driven acquisition,³³ neural network augmented imaging,³⁴ Fiji,³⁵ DeepImageJ,³⁶ CellProfiler,³⁷ Napari,³⁸ TrackMate,¹⁷ NucleAIzer,³⁹ CellPose,⁴⁰ StarDist,¹⁶ ilastik,⁴¹ TWS,⁴² Pytorch,⁴³ TensorFlow,⁴⁴ Keras,⁴⁵ scikit learn,⁴⁶ ZeroCostDL4Mic,⁴⁷ DeepCell,⁴⁸ Bioimage Model Zoo,⁴⁹ and Project Jupyter.⁵⁰

Table 1.

Examples of artificial neural networks used in biological image analysis

Feedforward neural network (FNN)	An FNN is a neural network architecture in which information is unidirectionally passed from the input layer to hidden layers and finally to the output layer. Each layer consists of several neurons, which perform mathematical operations on the inputs from neurons of the previous layer. These mathematical operations comprise linear transformations characterized by trainable parameters, followed by a nonlinear activation function, such as the sigmoid or rectified linear unit (ReLU) function.
Autoencoder	An autoencoder is a neural network architecture for unsupervised learning. It consists of two parts: an encoder and a decoder. The encoder maps the input to a lower-dimensional latent space, also called the bottleneck. The decoder then maps it back to the original input space. The autoencoder is trained to minimize the difference between the input and the reconstruction, while being unable to learn the identity function due to the imposed bottleneck. This general idea can be applied to a variety of computer vision tasks, including image segmentation (e.g., in FCNs) and denoising (e.g., in variational autoencoders, VAEs).
Convolutional neural network (CNN)	CNNs build on the concept of FNNs. Their defining elements are convolutional layers, which perform matrix multiplications on a local neighborhood.51,52 This allows them to capture spatial information, making them uniquely suited for image analysis. In 2012, a CNN called AlexNet53 won an image classification competition by a large margin, establishing CNNs as the state of the art for computer vision tasks and phenotype classification in biology.
Fully convolutional network (FCN)	FCNs, similarly to CNNs, also work on image inputs but yield an output image instead of a label. They have demonstrated superior performance in segmentation tasks54 and found immediate application for segmentation of challenging microscopy images55 as well as in image denoising. Many FCNs follow an encoder-decoder scheme, where a bottleneck is imposed to prevent the network from learning the identity and force it to learn a representation of the objects of interest.
Generative adversarial network (GAN)	GANs consist of two networks that are trained simultaneously in competition against each other.56 A generator network creates synthetic data (such as images), and a discriminator network predicts the probability of an image being synthetic or real. GANs have found application in image reconstruction and denoising, where, after training, the discriminator is dropped and only the generator is retained. It is also possible to constrain the generator on input data to obtain a representation of a specific image, leading to conditional GANs.57 This method has found application in image style transfer58,59 and in microscopy.47 GANs have also been employed for data augmentation, where synthetic images are created to increase the diversity in the training dataset that is used for subsequent network training.60
Vision transformer (ViT)	A vision transformer (ViT) is a neural network architecture that can be applied to several computer visions tasks, including object detection, classification, and segmentation.61 It is based on the transformer architecture,62 which was introduced in the field of natural language processing (NLP) and has since been adapted for image analysis.63 ViTs use self-attention mechanisms to focus on parts of the image with useful features and achieve state-of-the art performance on many benchmarks but are computationally expensive.

Table 2.

Overview of typical image analyses tasks in computer vision and their application in virological studies

Task	Example algorithms	Applications in virological studies
Segmentation	U-Net,55,64 FCNs, StarDist,16,65 NucleAIzer,39 Cellpose,40 Mesmer,66 Mask R-CNN67	detection of virus-specific antibodies,68 cell morphology,69,70 infection state assessment,70,71,72 screening for antiviral targets,73 viral immunosuppression,74 viral replication75
Classification	CNNs	infection state classification and trajectory prediction76,72,21
Tracking	CellCognition,77 TrackMate,17 DeLTA,78,79 DeepTrack80	hepatitis C virus tracking81,82
Denoising	CARE,83 N2S,84 N2N,85 Noise2Void2,19 structured N2V,86 probabilistic N2V,87 DivNoising,88 Blind2Unblind89	preprocessing for the analysis of single viral particles or for RNA-FISH
Image reconstruction	Deep-STORM,90,91 ANNA-PALM,34 DECODE92	Intracellular or subnuclear trafficking by SRM
AI-driven acquisition	event-driven acquisition33	viral egress

Object segmentation in fluorescence microscopy

Analysis of fluorescent microscopy images often begins with the segmentation of objects of interest. Classical segmentation provides image histograms, local changes in pixel intensity, or informs on the distribution of noise. It is implemented in algorithms such as Otsu thresholding, edge detection, or entropy-based thresholding methods.⁹³ While these approaches provide useful segmentation results for objects with high signal-to-noise ratio, accurate segmentation often presents a non-trivial problem due to confounding parameters. This is especially true for infected cells, which exhibit a high level of heterogeneity and low signal-to-noise ratio (Figures 1 and 3). Here, we present example algorithms that tackle this challenge.

Figure 3.

How ML enhances viral image analysis

(A) Nuclear segmentation of the Hoechst 33342 signal from human lung epithelial A549 cells infected with AdV-C2 expressing GFP from a cytomegalovirus promoter. The input image shows the Hoechst signal. Scale bar, 20 μm. Nuclear segmentation was performed using StarDist¹⁶ or minimum cross-entropy (MCE). For the segmented images, regions in green indicate correctly classified pixels and regions in magenta incorrectly classified pixels with respect to manually curated segmentation maps. White arrow points to an under-segmented nucleus in the StarDist protocol and an over-segmented nucleus in the MCE protocol. The intersection over union (IoU) of two images A and B is calculated as $\mathrm{I}\mathrm{o}\mathrm{U}(\mathrm{A},\mathrm{B})=\frac{\left|\mathrm{A}\cap \mathrm{B}\right|}{\left|\mathrm{A}\cup \mathrm{B}\right|}$, where $\left|\mathrm{A}\cap \mathrm{B}\right|$ is the area of overlap/intersection between A and B, and $\left|\mathrm{A}\cup \mathrm{B}\right|$ is the joint area/union of A and B. IoU values range between 0 and 1, with a higher value indicating a better congruence.

(B) Image classification using a convolutional neural network (CNN) as described in Andriasyan et al.²¹ Left image shows Hoechst 33342 staining of live A549 cells infected with AdV-C5. Scale bar, 20 μm. Center image shows classification of CNN, with nuclear masks for uninfected cells (yellow) and infected cells (red). Orange arrow points to an incorrectly classified nucleus and white arrow to a nucleus that was not segmented under the chosen probability threshold. The right image shows DAPI nuclear staining (blue) and immunofluorescence (IF) for the viral protein VI expressed late in infection. Of note, although the CNN was trained using a reporter virus that expresses GFP under control of the early-to-late viral IX promoter, the network recognizes features of cells positive for the viral late protein VI indicating network robustness.

(C) Image denoising using Noise2Void2.¹⁹ A549-ΔMIB-1 cells were infected with genome-tagged AdV-C5-EdC as described.^20,18 Images show maximum intensity projections of confocal stacks of EdC staining (image courtesy of Alfonso Gómez-González). The scale bars indicate 5 μm for the full-size image and 1 μm for the zoomed-in image. Image smoothing by Gaussian filtering can reduce pixel noise but is less efficient in removing background noise or enhancing the signal from the object of interest. For more comprehensive discussions of image analysis algorithms, metrics, and benchmarking, see Maier-Hein et al., Ulman et al., and Caicedo et al.^94,95,96

Early ML frameworks for microscopy image segmentation, such as Trainable Weka Segmentation (TWS)⁴² and ilastik⁴¹ treat segmentation as a pixel classification problem (Figure 2). Through a graphical user interface, the user initially marks pixels that belong to a given class, e.g., cell or background. In a next step, a set of features is selected, e.g., intensity, texture, shape, or noise, and an ML model is then trained for object segmentation. TWS and ilastik both rely on decision trees or the related random forest algorithm by default, although other ML models are also available. Decision trees follow a comprehensible, rule-based process but have limitations in incorporating global image context or addressing subtle object features. Thanks to their simplicity and user-friendliness, TWS and ilastik continue to be used in microscopy image analysis, although they are increasingly complemented by other tools.

An important breakthrough in addressing segmentation problems was the introduction of a feedforward NN (FNN) architecture, called fully convolutional network (FCN, see Table 1).⁵⁴ FCN maps an input image to a segmented image. In a supervised learning setting, FCNs are provided with the raw images along with the desired segmentation maps, which are often manually curated. One important building block of FCNs are the convolutional layers, which perform mathematical operations on a local pixel level using trainable parameters and thereby extract image features, for example object morphology. Subsequently, deconvolution layers decode the learned representation in a process also called backward convolution, yielding a final segmentation map. A seminal implementation of this approach for biomedical images has been U-Net.⁵⁵ U-Nets can learn semantic segmentation, e.g., discrimination between foreground and background by minimizing the cross-entropy loss between the provided ground truth object masks and the predicted segmentation map. They use skip connections between the encoding and the decoding path, which allows the network to retain the correct object positions. An additional pixel-wise loss penalizes misclassifications at object border regions and incentivizes the network to learn correct instance segmentation, that is, the discrimination between objects of the same class. However, the correct delineation of objects from each other remains a challenge,⁹⁷ which may be mitigated by post-processing, such as using watershed transformation.

For domain-specific solutions, the U-Net architecture has already been improved upon in many different imaging modalities.^64,98 One example is the StarDist algorithm.¹⁶ It proved to be effective in accurately segmenting and delineating (declumping) overlapping cells and nuclei in complex biological specimens at multiple modalities. It can, for example, cope with large variations in object brightness (Figures 1A and 3A). The developers of the algorithm assume that the objects of interest have circular shape and, more precisely, that they can be represented by star-convex polygons. By incorporating prior knowledge about the typical object shape, this approach provides a much better approximation to a real object than rectangular bounding boxes commonly used in photorealistic images, as described earlier.⁶⁷ StarDist learns to associate the object probability of each pixel with object distance from the nearest background pixel, thus favoring pixels in the object center. In addition to object probabilities, the algorithm predicts a set of radial distances for polygonal object approximation. Finally, conflicts of overlapping objects are resolved by non-maximum suppression to yield a segmentation map with multiple object instances. More recently, this algorithm was extended to 3D segmentation by utilizing star-convex polyhedra,⁶⁵ and it continues to achieve state-of-the-art performance.⁹⁹ It has been plugged into Fiji³⁵ and CellProfiler¹⁰⁰ and is frequently used in virology, for example, to study the effects of antibodies against viral glycoproteins in cell-to-cell transmission of simian foamy virus or SARS-CoV-2.^68,69,101

NucleAIzer is a segmentation framework that utilizes a generative adversarial network (GAN) for style transfer in order to automatically create enhanced samples during training³⁹ (see Table 1). This approach makes DL straightforward for nuclear segmentation, since it readily identifies nuclei in diverse experimental settings. Cellpose⁴⁰ and Mesmer,⁶⁶ on the other hand, are more general segmentation frameworks and allow for efficient segmentation of diverse objects. They have been trained on comprehensive sets of images from cells grown in 2D cultures and provide resources for fine-tuning algorithmic output.¹⁰² Intriguingly, Cellpose also segments 3D images⁴⁰ and is attractive for 3D organoid models or air-liquid interface cultures relevant for oncolytic therapy or respiratory pathogen infections.^103,104

A more recent development is represented by the so-called transformer models. Originally introduced for natural language processing,⁶² these models include GPT-3 and -4 and excel at a wide range of tasks and likely will have a fundamental impact on science.^105,106 Vision transformers (ViTs) have been adapted for image analyses.⁶³ They combine convolutional layers with self-attention mechanisms to weigh the relevance of parts of the input image and achieve state-of-the-art performance on a wide range of computer vision tasks, including image segmentation.⁶¹ One recent example of a powerful ViT-based model is the Segment Anything Model, which was trained on a collection of more than 11 million images.¹⁰⁷ It generalizes to a variety of segmentation tasks, including objects not encountered during training. Compared to convolutional neural networks (CNNs), the demand for comparatively powerful hardware in training and inference makes a wide distribution of ViT models computationally expensive. Although ViTs have not yet been used in virological image analyses, their adaptation to virology is just a matter of time, as supervised ML already facilitates a wide range of segmentation tasks in virology. This includes the quantification of syncytia formation by the SARS-CoV-2 receptors angiotensin converting enzyme 2 and neuropilin-1,⁷⁰ segmentation of SARS-CoV-2-infected cells in CRISPR screening,⁷³ quantitative viral gene expression studies,^71,73,75 or lentivirus-dependent immunosuppression in tissue microenvironment.⁷⁴

Virus tracking in live cells

Time-resolved particle tracking analyzes the trafficking of virus particles in live cells and informs on infection and nanoparticle delivery mechanisms.¹⁰⁸ Akin to image segmentation, tracking requires the combination of object detection and linkage over time. The first quantitative virion trafficking experiments in cultured cells used manual tracking of sparsely seeded fluorescently labeled AdV.¹⁰⁹ Several years later, feature point-tracking methods were introduced,¹¹⁰ which allowed for automated detection of virions and analyses of their trajectories on the cell surface, for example.¹¹¹ It opened the field to implement mechanical cues acting on invading virus particles and initiating virion uncoating.¹¹² The challenge of handling fluctuating objects owing to photobleaching or movement in and out of the focal plane can be addressed by a two-step solution to the linear assignment problem and provides a framework to efficiently link objects across time.¹¹³ The introduction of automated tracking software, such as CellCognition, allowed tracking of complex objects using a nearest-neighbor algorithm and enabled accurate object annotation in morphologically distinct biological environments, including mitotic cells.⁷⁷ A more recent framework is TrackMate, which offers a high flexibility in choosing segmentation, tracking, and linkage algorithms, on top of being compatible with StarDist.^17,114 It is available as a Fiji plugin and has been employed to automatically track infected cells in time-lapse movies or to investigate changes in the localization of viral entry receptors.^115,116

Recent work has also shown that tracking can be accomplished by different DL methods, which are well suited to cope with challenging segmentation tasks, high object density, occlusions, or sudden changes in object trajectories. Examples for DL-based tracking frameworks include DeLTA,⁷⁸ DeepTrack,^117,118 DeepTrack2,⁸⁰ and MAGIK.¹¹⁹ The DeepTrack framework uses a U-Net for object segmentation and tracks by conventional procedures, akin to DeLTA. Meanwhile, MAGIK builds on the concept of graph neural networks, which are increasingly used in biology. Some DL tracking algorithms have already been tested with viral particles, such as hepatitis C virus, and represent promising avenues for virological image analysis.^81,82

Classification of objects in fluorescence microscopy

Once objects of interest have been identified and segmented, the next step is often to perform a deeper phenotypic characterization, for example a correlation of information from the images with infection state (Figures 1D and 3B). This image classification is frequently achieved by an FNN called CNN (see Table 1). Compared to FCNs used for image segmentation, CNNs contain dense (fully connected) layers, which map the image features to a class label instead of a segmentation map. Proper tuning of CNNs requires expert annotation of training datasets, where images are presented together with their corresponding labels in supervised learning. CNNs were successfully used to detect Zika virus infection in U87 glioblastoma cells based on changes in the morphology of the endoplasmic reticulum.⁷⁶ Similarly, CNNs proved useful in characterizing phenotypes of dermal fibroblasts infected with human cytomegalovirus (HCMV), and showed how HCMV induces nuclear reorganization to enhance viral genome replication.⁷² Using the nuclear and viral assembly compartment signal, the authors trained a mask region-based CNN (Mask R-CNN)⁶⁷ and a residual neural network-32 (ResNet-32)¹²⁰ for object detection and infection state classification, respectively. Other CNNs identified herpesvirus- or AdV-infected cells stained with a generic double-stranded DNA-binding dye in absence of virus-specific information.²¹ Extension of the underlying CNN was then used to predict cell lysis and release of virus for cell-free transmission to neighboring cells. Reassuringly, the trained model was transferable between different microscopy modalities, indicating robustness and broad applicability.

Image classification can be performed both after and during the acquisition process (Figure 2). One example is event-driven acquisition in live cell imaging, where images are directly passed to a neural network for analysis, and feedback is used on the fly to adjust the acquisition rate of the microscope.³³ This type of dynamic imaging is particularly attractive when monitoring rare, fast, or irreversible processes, such as virus-induced cell lysis. It enables studies of difficult problems in virology, such as predictive distinction between lytic and non-lytic modes of virus transmission.²¹

Extending the limits of super-resolution microscopy

Most virus particles have a diameter around 20–200 nm, below the diffraction limit of conventional light microscopes. To address this limitation, computational methods have been developed to improve the resolution of diffraction-limited objects, hand in hand with improvements of hardware and optics. Several experimental approaches exploited nonlinear or stochastic effects in optical systems to overcome the diffraction limit. These methods are referred to as SRM and include structured illumination microscopy (SIM) and stimulated emission depletion (STED), as well as various single-molecule localization microscopy (SMLM) methods, such as photoactivated localization microscopy (PALM) and stochastic optical reconstruction microscopy (STORM). For a more detailed discussion, the reader is referred to Schermelleh et al.⁹

SRM has enabled scientists to study single virus particles and subviral structures in great depth. For example, PALM and STED microscopy have shed light on the distribution of human immunodeficiency virus 1 (HIV-1) Gag protein at the plasma membrane, showcasing the dynamics of viral surface proteins in particle maturation.^121,122 Work by Wang and colleagues showed that single DNA genomes of herpes simplex virus type 1 (HSV-1), AdV, and vaccinia virus can be tracked in infected cells by gated-STED, enabling a detailed analysis of viral DNA delivery and replication in the nucleus or the cytoplasm.²⁰ STED microscopy was also combined with DL to reveal Zika virus-induced changes of endoplasmic reticulum morphology favoring viral replication.⁷⁶ Others used STORM to elucidate the spatial organization of HSV-1 proteins in the viral tegument¹²³ or SIM combined with ML-based particle morphology analysis to dissect the assembly processes of Newcastle disease virus and influenza virus particles.¹²⁴

Most SRM methods are compatible with live cell microscopy, but their practical application can pose considerable challenges. This is because increased resolution by SRM often comes at the expense of lower contrast and exacerbated phototoxicity, a particular challenge in live cell experiments. Since many SRM techniques, such as SMLM or SIM, capture several images of the same biological structures and rely on computational reconstruction for the synthesis of the final image, some of the problems of SRM can be mitigated by improvements in image processing. Indeed, ML algorithms are useful for image reconstruction, deconvolution, and denoising (Figures 2 and 3C). They allow for a reduction of illumination time, laser power, or number of images for reconstruction, and they reduce computational cost. Recently, new algorithms have been developed for application in STORM, PALM, SIM, light field microscopy, and other SRM procedures.^{34,90,91,92,125,126,127} They have already found broad applications in cell biology and are awaiting implementation in virological research.

Enhancing conventional light microscopy

Besides overcoming the diffraction limit by sophisticated SRM setups, computational algorithms focus on improving the resolution of conventional light microscopy systems. This is of particular interest to virology, as many virus particles are smaller than the diffraction limit of conventional light microscopes, and hence improved resolution would benefit their subcellular analyses. Here, we discuss three approaches, deconvolution, supervised learning from pairs of high-/low-resolution images, and unsupervised/self-supervised learning. The first approach recovers object details by inverting the image formation process, which requires sufficiently accurate knowledge of the point spread function (PSF). The PSF describes how an optical system forms an image of a point source. One computational algorithm to invert the image formation process is the Richardson-Lucy deconvolution^128,129 Of note, this approach is derived from first principles, which makes it seemingly compelling, although it makes simplifying assumptions about the nature of the noise and the isotropy of the PSF. Recent work showed that employing DL to combine the Richardson-Lucy deconvolution with an FCN leads to improved deconvolution performance and lower susceptibility to artifacts caused by out-of-focus light.¹³⁰ Another method motivated by physics has been super-resolution radial fluctuations, which exploits oscillations in emitter intensity over time and allows the determination of the fluorophore position with increased precision.^131,132

Secondly, DL has been used to learn a mapping between low- and high-resolution images, which is referred to as single-image super-resolution. The approach was implemented, for example, in content-aware image restoration,⁸³ where the authors could reconstruct high-resolution images from under-sampled images by employing an FCN based on the U-Net architecture.⁶⁴ While such DL-based approaches risk introducing artifacts during the image reconstruction process, this problem can be mitigated by image quality assessment. For example, imaging artifacts that occur in the reconstruction process can be evaluated by SQUIRREL.¹³³ This algorithm operates on the assumption that the conventional and the super-resolution images contain the same biological structure, and their comparison can identify artifacts. Similarly, the perceived quality of the reconstructed image can be assessed by metrics, such as the multi-scale structural similarity index measure,¹³⁴ which can be incorporated into the loss function during network training.

While the previous procedure relies on pairs of low-/high-resolution images for network training in a supervised setting, the third approach has explored image enhancement without high-resolution images. This is mostly used for image denoising and employs DL to learn properties of the image noise. Example algorithms are Noise2Noise,⁸⁵ Noise2Self,⁸⁴ Noise2Void2,¹⁹ probabilistic Noise2Void,⁸⁷ and DivNoising⁸⁸ (Figure 3C). Many of these algorithms assume that the image noise is pixel-wise independent, but the signal from the object of interest is not. This allows network training in the absence of ground truth with intriguing results approaching the accuracy of the supervised methods described above. Some extensions of these algorithms can also take global context into account,⁸⁹ estimate the uncertainty of the pixel-wise noise,⁸⁷ or cope with structured noise often found in practice.⁸⁶ It is important to keep in mind, however, that none of these methods can unambiguously recover a clean image. This notion is exemplified by DivNoising,⁸⁸ which builds on the concept of variational autoencoders (see Table 1) and generates several plausible versions of a denoised image.

To date, applications of denoising algorithms in virological studies have remained scarce. In addition to the computational complexity, one problem has been the difficulty to collect ground truth data for validation. Such data can be obtained by acquisition of matched image pairs with high/low noise levels or by simulation. In addition, improvements in segmentation and classification algorithms discussed above may cope with certain levels of noise and alleviate the need for preprocessing.

Applications in electron microscopy

The introduction of the transmission electron microscope (TEM) to the life sciences by Max Knoll and Ernst Ruska in the early 1930s enabled scientists to visualize and understand the ultrastructure of viruses and other pathogens. TEM has since been extensively used to examine virions, such as tobacco mosaic virus, poxvirus, herpesvirus, HIV, or AdV. In the 1950s, it was enhanced and greatly simplified by negative staining protocols using heavy metal salt solutions, such as phosphotungstic acid or uranyl acetate.¹³⁵ This helped to delineate the contours of virions and viral capsomers and the description of physical principles in virus structure and assembly.^136,137 In addition, it readily revealed characteristic features of virus families, such as naked proteinaceous capsids of icosahedral symmetry, lipid envelopes, viral spikes embedded in the envelope, or the discovery of mRNA splicing.^{138,139,140,141} Flash-freezing protocols for cryogenic electron microscopy (EM) that vitrify water and preserve the structure of biomolecules in cells and viruses enabled investigations of biological samples at atomic resolution.^{142,143,144,145,146,147} Nowadays, AI-based structure prediction increasingly complements high-resolution EM imaging of cell structures, besides viruses, and enhances the reconstruction of previously intractable macromolecular complexes, such as the nuclear pore complex.^148,149

EM image analysis problems can in principle be tackled using the same approaches as in fluorescence microscopy. However, due to the lack of probes with molecular specificity and the destructive nature of high-energy electron beams, EM images typically have lower contrast and require significantly more effort for data annotation and curation than typical fluorescence images. Some procedures have successfully addressed this challenge by employing an ensemble of AI models. For example, several CNNs automate segmentation of 3D EM images.¹⁵⁰ Other work generated annotated 3D maps of focused ion beam scanning electron microscopy data by using an ensemble of methods, including 3D U-Net,¹⁵¹ 3D-StarDist,⁶⁵ ilastik,⁴¹ and manual annotation.^152,153 Others developed a CNN to aid segmentation of subcellular structures in cryogenic electron tomography (cryo-ET),¹⁵⁴ which is available as part of EMAN2.¹⁵⁵ The CNN-based predictions facilitated annotation of cryo-ET images and were used in multiple virological studies, including cell egress of SARS-CoV-2¹⁵⁶ and Chikungunya virus.¹⁵⁷ These efforts have recently been combined with two CNN-based frameworks for multi-class segmentation, namely DeepFinder¹⁵⁸ and DeePiCt,¹⁵⁹ and represent the state of the art in the field.

Dedicated algorithms have also been developed for virological applications. For example, a U-Net-based approach accurately and quickly segments and classifies viruses in EM micrographs.¹⁶⁰ The authors provide useful methodology for training robust models. Yet others developed tools for detection and classification of viruses in EM images.^161,162 Underlying CNNs have been instrumental to analyze the secondary envelopment of HCMV, a critical step in the morphogenesis of infectious HCMV particles.¹⁶³ Additional work has developed a CNN to automatically recognize cytoplasmic HCMV capsids in TEM images and help dissect the HCMV capsid envelopment process.¹⁶⁴ The authors addressed the issue of limited training data and low performance by employing a GAN to augment the expert-labeled training dataset. With this, they demonstrated that their low-cost strategy increased detector performance and facilitated the interpretation of viral EM images.

Conclusion and perspectives

A major quest in viral and biological imaging has been to provide a basis for causal understanding of spatiotemporal patterns. A classical approach uses mathematical formulations of dynamic biological patterns based on physical principles, for example the seminal analyses of soft matter interactions, advection, diffusion, and reaction kinetics in actin dynamics.¹⁶⁵ A virological example of an early white box infection model was the mathematical simulation of cell lysis by AdV,¹⁶⁶ an agent widely used in gene delivery.¹⁶⁷ In many cases, white box models have been difficult to establish, however, due to the scarcity of knowledge of the underlying physical parameters, and because the information available in patterns cannot be merely extracted from individual pixels but rather requires macroscopic contextual-level understanding and an experienced interpreter.^168,169,170

On the other hand, AI-guided computational microscopy is reaching a stage where multiple objectives can be addressed, including object detection, systematic extraction of image features, image restoration, and object quantification. In fact, AI has the potential to fuel microscopy-based virological research by outperforming conventional methods in image processing and analysis, such as deconvolution, segmentation, and classification (Figure 3). An important further advance has been that the image label scarcity for supervised learning can be mitigated by reducing the annotation time using active learning,¹⁷¹ noisy student training,¹⁷² weak annotation,¹⁷³ semi-supervised learning,¹⁷⁴ few-shot learning,¹⁷⁵ zero-shot learning,¹⁷⁶ or test-time augmentation.¹⁷⁷ This increasingly demonstrates that AI is reaching a stage of maturity where it can extract information from a variety of data types and address problems in a logically consistent manner.¹⁰⁶

Furthermore, high-dimensional microscopy datasets will benefit from statistical methods and unsupervised learning. They will help to uncover complex interconnections between variables and formulate testable hypotheses. An example here is the study of simian immunodeficiency virus-induced immunosuppression in lymphoid tissue, where the authors first performed cell segmentation and feature extraction and then used self-organizing maps¹⁷⁸ to identify different cell (pheno)types.⁷⁴ Another example is hypothesis generation by ML as provided by the CellPainting protocol¹⁷⁹ with multiplexed image-based profiling of subcellular structures.^180,181 Hierarchical clustering of features groups cells with similar phenotypes and can reveal similarities between affected pathways in high-content imaging drug screens (compare Figure 1E). Recent work has extended this approach and showed that segmentation, feature extraction, and clustering can be substituted by CNNs.^182,183 Notably, this protocol has found application in virological research, for example an antiviral drug screen against coronavirus 229E¹⁸⁴ or the pathway underlying SARS-CoV-2 inhibition by 4-octyl-itaconate.¹⁸⁵

Regardless of the promise of AI, any conscientious application of DL algorithms requires attention to possible bias. Bias may be hard to reconcile, however, due to the complex nature of the computational network architectures. Firm conclusions from AI-enhanced data may thus be drawn only if results can be validated by orthogonal approaches. One way to validate AI-based output is to use physical models promoting the concept of explainable AI.¹⁸⁶ Validation becomes increasingly important not only in image analysis but also biology and personalized applications in medicine and society at large. In summary, the potential of combining imaging and computational image analysis is immense. It holds promise to ease the transition from 2D cell cultures, 3D organoids, and animal models toward accurate clinical diagnosis and curative treatment of disease.^187,188