Automated and semi-automated enhancement, segmentation and tracing of cytoskeletal networks in microscopic images: A review

Abstract

Cytoskeletal filaments are structures of utmost importance to biological cells and organisms due to their versatility and the significant functions they perform. These biopolymers are most often organised into network-like scaffolds with a complex morphology. Understanding the geometrical and topological organisation of these networks provides key insights into their functional roles. However, this non-trivial task requires a combination of high-resolution microscopy and sophisticated image processing/analysis software. The correct analysis of the network structure and connectivity needs precise segmentation of microscopic images. While segmentation of filament-like objects is a well-studied concept in biomedical imaging, where tracing of neurons and blood vessels is routine, there are comparatively fewer studies focusing on the segmentation of cytoskeletal filaments and networks from microscopic images. The developments in the fields of microscopy, computer vision and deep learning, however, began to facilitate the task, as reflected by an increase in the recent literature on the topic. Here, we aim to provide a short summary of the research on the (semi-)automated enhancement, segmentation and tracing methods that are particularly designed and developed for microscopic images of cytoskeletal networks. In addition to providing an overview of the conventional methods, we cover the recently introduced, deep-learning-assisted methods alongside the advantages they offer over classical methods.

1. Introduction

Imaging is an indispensable tool for discovery/diagnostics in biology and biomedicine. It is also a fast-evolving field, with frequent emergence of novel techniques that focus on specific questions. Bioimaging techniques can be broadly categorized into two groups: i) optical and electron microscopy-based techniques for micro- and nano-scale imaging, and ii) magnetic resonance (MR)-based techniques for tissue- and organ-scale imaging. These techniques are further sub-branched into multiple modalities, each depending on its own set of instruments and software. Consequently, the different modes of image acquisition generate images with inherently different features (e.g., point spread function, noise distribution, spatiotemporal resolution). These acquisition-based differences, as well as the diverse structural and dynamic properties of the biological specimens have restricted the widespread adoption and utilisation of generic methods in bioimage processing/analysis. Rather, researchers often need to devise individual project- and question-based solutions for their analyses. Automation has therefore been an important challenge in the field of bioimage analysis.

Automation is particularly important for image segmentation, a technique that is central to many image analysis pipelines. Manual segmentation of objects from images is tedious and prone to user-to-user variability. Recent advances in the field of computer vision have made an impact on bioimage analysis with the introduction of a number of segmentation tools that found widespread use among biologists. These tools have a higher degree of automation compared to classical segmentation methods owing to the higher extent of generalising power they acquire through training with large amounts of image data.

In the process of designing a segmentation pipeline, the structural properties of the objects to be segmented play an important part. Particularly in the field of cell biology, the redundancy of certain morphological features (blobs, tubules, vesicles, etc.) prompts scientists to develop automated object detection and segmentation algorithms that are tailored to capture these patterns. Curvilinearity is one such redundant morphology. Networks of curvilinear objects are ubiquitous in biology, ranging from the cytoskeletal filaments at micro- and nano-scale to blood vessel networks at tissue/organ level. The prevalence of this type of morphology across scales in biological systems suggests that it provides certain functional advantages. As a result, one focus of the image processing efforts in biology and biomedicine is on the automation of segmentation and tracing of curvilinear structures. In the context of vascular and neuronal networks, a large body of literature addresses this topic (reviewed in [1], [51], [59], [29], [71]). The algorithms and methods developed in these studies are in principle applicable to microscopic images of cytoskeletal filaments due to their similar morphology. In practice, however, the microscopic images of cytoskeletal networks often require more customised techniques, since i) the different imaging techniques and modalities generate different signal distributions, ii) the cytoskeletal filament thickness is smaller than the resolution limit of most microscopy methods, a point that has to be addressed during the segmentation.

In this regard, this review focuses on recent literature regarding segmentation/tracing cytoskeletal filament networks in microscopic images. While the focus is on the segmentation/tracing, we also discuss a few studies covering other related image-processing tasks such as vesselness-enhancement, time-tracking, morphological network analysis and handling of microscopic limitations. The review starts with a section introducing the different microscopy categories that are frequently used for imaging of cytoskeletal filaments. In the subsequent section, different categories of segmentation are briefly explained. The review then continues with two sections introducing some of the recent research on the classical and deep learning-based methods for enhancement and segmentation of cytoskeleton, before concluding with a section for summary and outlook.

2. Categorisation of microscopy methods

Imaging of cytoskeletal filaments is performed using a variety of microscopy methods, ranging from fluorescence microscopy to cryo-electron tomography. Each of these methods produces images with different spatiotemporal resolution, signal-to-noise ratio and contrast properties. Studies targeting particular aspects of cytoskeleton structure and function must first choose a microscopy method capable of providing that information. For instance, a study aiming for a very high resolution can resort to an electron microscopy method, which can achieve a spatial resolution of around 5 nm. Electron microscopy images, however, typically have very low signal-to-noise ratio and low contrast, which makes visualisation, processing and analysis of these images difficult, especially in 3D [36]. Other studies, which focus on the dynamics of the filaments, or co-localisation of filaments with particular other proteins, often apply fluorescence-microscopy-based methods (e.g. confocal and widefield systems). One advantage of these methods is their speed, which enables them to be applied in a high-throughput manner [53]. Speed of imaging is also particularly important for capturing the cytoskeletal dynamics from time-series datasets. A major limitation of the diffraction-limited fluorescence microscopy in general is its low resolution and the artifacts/blur associated with the out-of-focus fluorescence. These limitations are particularly aggravated in widefield systems, where the out-of-focus fluorescence is not filtered at all. By rejecting the out-of-focus fluorescence via a pinhole, laser-scanning confocal microscopy (LSCM) can achieve better contrast and resolution (roughly 250 nm in xy and 750 nm in z), albeit at the cost of imaging speed. A particular limitation of the LSCM is that repetitive irradiation of the specimen with the excitation light during the image acquisition increases the chances of phototoxicity (with live cells) and photobleaching - which reduces the signal-to-noise ratio in the output images. Improvements can be gained to achieve a better trade-off between speed, sensitivity and resolution using more specialised set-ups such as spinning disk system or widefield deconvolution microscopy. A special case is Total Internal Reflection Fluorescence Microscopy (TIRFM), a method that exploits evanescent wave for the selective excitation of fluorophores that are very close to the coverslip. TIRFM offers improved contrast and reduced photodamage due to the restricted area of excitation, which eliminates the signal from the bulk of the cell. This makes the technique advantageous for time-series imaging. It is, therefore, frequently used for studying cytoskeleton dynamics near the plasma membrane [85], [55]. None of the diffraction-limited methods, however, are capable of a resolution that permits the precise localisation of the cytoskeletal filaments, which have a lateral width (24 nm for microtubules, 7 nm for actin filaments) far beyond the resolution limit of these techniques. This poses an important challenge for the segmentation and the subsequent quantitative analysis of the cytoskeleton images acquired via these methods.

An ensemble of techniques called super-resolution techniques can overcome the diffraction barrier to achieve better resolution [73]). Common super-resolution methods include Stimulated Emission Depletion Microscopy (STED), Single Molecule Localisation Microscopy (SMLM) and Structured Illumination Microscopy (SIM). STED is an optics-only approach that can in principle achieve about 20 nm resolution in xy and 50 nm in z. However, STED requires irradiation of the specimen with very high laser power, which raises serious phototoxicity issues with living cells, especially in time-series imaging, which necessitates prolonged irradiation. SMLM is an umbrella term for several super-resolution techniques using a common principle [73], [37], including Photo-Activated Localisation Microscopy (PALM), Stochastic Optical Reconstruction Microscopy (STORM) and DNA-PAINT methods. These methods are not optics-only; rather, the super-resolution image is reconstructed from the “point clouds”, coordinates of sparse molecular localisations that are deduced from sequential imaging of random subsets of fluorophores. The image reconstruction from the localisation data is achieved using complex postprocessing algorithms. SMLM methods can typically achieve a resolution of 10–30 nm, and thus can achieve nearly-precise localisation of cytoskeletal filaments. However, the successful reconstruction of super-resolved images requires acquisition of extremely large numbers of frames [26]. This is an inherently slow process that hinders the applicability of these techniques for live-cell imaging and tracking of dynamic processes such as growth/shrinking of cytoskeletal filaments. Long illumination times also increase the chances of photobleaching. Ongoing research strives to develop algorithms that can reconstruct the super-resolved image from increasingly sparser localisation data, hence speeding up the process [63]. SIM is a versatile category of the super-resolution methods, which offers faster image acquisition. SIM utilises non-uniform, patterned illumination of the specimen. The interactions between the high-frequency illumination pattern and the high-frequency variations in the specimen produce what is called a Moiré interference pattern, which contains the super-resolution information. A series of images with Moiré patterns are thus collected, changing the orientation of the illumination with each image. The collection of images are then processed with specialised algorithms to reconstruct the super-resolved image. Despite its similarity to SMLM in terms of the dependency on the acquisition of multiple image frames and their digital postprocessing, SIM is much faster because it requires much fewer raw images (typically 9 in 2D and 15 in 3D), and thus facilitates time-series imaging. On the other hand, SIM resolution is typically around 100 nm in xy and 250 nm in z, which is not high enough for precise localisation of cytoskeletal filaments.

3. Categorisation of segmentation methods

Segmentation is the process of partitioning an image into multiple meaningful regions based on certain coherence criteria. The output of the segmentation process is an image, where each pixel/voxel is replaced by a label, which is determined based on these criteria. In this respect, image segmentation is a pixel/voxel classification problem.

Segmentation methods can be categorised based on the goal of this classification process. Binary segmentation simply aims to partition an image into foreground and background regions; thus the aim of classification is simply to distinguish the foreground pixels/voxels from the background. This is the simplest segmentation method, and the most common approach for segmentation of microscopic images. Very often, the binarised microscopic image is subjected to a “connected component analysis”, whereby the isolated foreground regions are given specific labels. Each labelled region can then be analysed separately. This approach to obtaining a multi-labelled image, however, is naïve due to two main reasons. First of all, if any objects in the binary image overlap with each other, these objects cannot be isolated based on connectivity, and thus cannot be properly labelled. Second, even if the objects can be properly labelled, these labels do not carry any information about what the labelled objects represent, but only report that the objects are spatially disconnected from each other. However, an analyst often wishes to categorise objects based on other criteria, rather than merely spatial connectivity. Certain segmentation algorithms tackle this problem by exploiting prior knowledge about the objects of the image under investigation. This group of segmentation methods, called semantic segmentation, are able to partition an image into semantically meaningful regions[58]. In other words, these methods allocate the pixels/voxels into pre-defined categories of object identity so that each label in the segmented image reports the class-membership of all pixels/voxels lying under that label. While being able to categorically identify different objects in an image, semantic segmentation still lacks the ability to distinguish different object instances belonging to the same category. Instead, multiple different objects falling into the same category are treated as a single entity and given a single label. This problem is addressed by another class of methods referred to as instance segmentation [58]. These methods are capable of distinguishing the different object instances belonging to the same category. Instance segmentation is particularly important for analysis of time-series microscopy images, because the ability to recognise and delineate objects at the instance level can substantially improve the performance of time-tracking of individual objects. Panoptic segmentation [38] unifies semantic segmentation and instance segmentation, assigning a categorical label to each pixel/voxel while also distinguishing different instances of each category.

Segmentation methods, therefore, are different approaches to solving the aforementioned classification problems. Based on the requirement for annotated data, segmentation methods can be divided into two broad categories: unsupervised and supervised segmentation methods. Unsupervised methods need no training data to segment images, whereas supervised methods fit a segmentation model based on a ground-truth dataset, which comprise examples of segmentations, usually created via manual annotation by domain experts.

Most of the classical segmentation methods belong to the category of unsupervised methods. These can be roughly divided into the following subcategories i) thresholding, ii) edge-based, iii) region-growing, iv) clustering, v) model-based. Thresholding methods aim to automatically find the optimal intensity threshold to divide an image into foreground and background regions. Edge-based methods seek to segment the image based on the detection of object boundaries. Region-growing methods are based on the assumption that pixels/voxels localised closely to each other are likely to belong to the same object. Clustering methods utilise various statistical clustering techniques to group pixels/voxels. Model-based methods typically solve a constrained optimisation problem to segment the image.

Supervised segmentation strategies are mostly based on deep neural networks (DNNs, [77], [69], [9], [17], [28]), although, more seldom, other machine learning algorithms are also used for this purpose [76], [2]. Ground-truth images are obtained either through direct manual annotation of the images, or manual correction following a rough segmentation via an unsupervised approach. Supervised learning itself is also further subcategorised based on the degrees and types of supervision. In the frame of image segmentation, strong (or full) supervision refers to the labelling of each pixel/voxel of each image in the training dataset. This is a time- and labour-intensive process. Efforts to tackle this particular problem often employ the semi-supervision strategy, which involves partial labelling of the pixels/voxels in the training dataset. As an example, Çiçek et al. [18] develop a neural network that can learn volumetric segmentation from 2D annotated slices. Another supervision mechanism, even further reducing the data-labelling workload is termed weak supervision. For an excellent review on the concept of weakly supervised learning in general, we refer the reader to Zhou et al. [96]. In the context of supervised image segmentation, a weakly annotated image can be obtained by marking individual objects in the image via bounding boxes, scribbles, points or at image level [95]. The weakest form of supervision is the image-level supervision, where the image is tagged for the presence/absence of a particular object category without specifying the localisation [95]. For an in-depth discussion and comparative evaluation of weakly supervised segmentation methods in different image domains, we refer the reader to Chan et al. [16].

In the field of cytoskeleton image analysis, segmentation tasks are usually more complicated than simple binary segmentation. Many studies aim to detect and extract the centrelines (or ridges) of the filaments in an accurate manner [91], a process termed filament tracing. Moreover, segmentation/tracing must often be performed on time-series images in order to facilitate a quantitative analysis of the growth/collapse dynamics of cytoskeletal polymers. This type of analysis requires techniques for the detection and labelling of each different filament in a network and the tracking of the individual labels over time. This task is also studied in the frame of instance segmentation [54]. Therefore, a wide spectrum of segmentation methods, ranging from the traditional threshold-based methods to the complex deep learning approaches, are utilised for the segmentation of cytoskeletal filaments. These are also accompanied by image pre-processing/enhancement strategies that are helpful (and often required) for the segmentation process. Below, we will summarise some of the recent studies focusing on these methods.

4. Classical methods for enhancement and segmentation of cytoskeletons

4.1. Conventional segmentation methods

Many of the conventional segmentation methods typically involve two steps: i) a pre-processing/feature enhancement step that denoises the image and enhances certain geometrical features of the objects to be segmented, and ii) a labelling step that groups the voxels into different categories based on the features extracted in the first step. In case of curvilinear structures, the pre-processing step aims particularly to enhance the vessel-like structures in the image using specific image filters, while suppressing structures that deviate from this curvilinear geometry. These vessel-enhanced images are then used for extraction of the filaments in the second step of the segmentation, which, in the simplest case, involves intensity-thresholding of the filtered image to obtain a binary mask of the curvilinear structures. An example of this approach is used by Alioscha et al. [3], who segment actin filament networks from fluorescence microscopy images. Their method starts with an image decomposition operation, which yields a cartoon image component and a noise/texture image component. The cartoon component is then used as input for the computation of a multiscale line-response image (via a method proposed originally by [61], where each pixel holds a score of belonging to a line. The authors then threshold this response image using a local adaptive thresholding method [86] to segment the actin network. Similar studies use filtering methods suggested originally by Sandberg & Brega [72], called line filter transform (LFT) and orientation filter transform (OFT). Among these, Zhang et al. [94] develop a software tool (SIFNE) for extraction of filament networks from images acquired via Single Molecule Localisation Microscopy (SMLM), a category of super-resolution microscopy techniques. In the segmentation stage of their analysis pipeline, they transform the images with LFT and OFT before extracting the binary mask from the vessel-enhanced images via Otsu thresholding. Finally, the binary images acquired are skeletonised via morphological thinning. Xia et al. [87] develop a method for quantitative analysis of cortical actin network using images acquired via STORM (Stochastic Optical Reconstruction Microscopy), a subcategory of SMLM imaging. Their segmentation involves LFT and OFT filters to enhance the curvilinear features in the image, followed by H-minima transform for further denoising, and finally binarisation of this pre-processed image via Meyer watershed transform [57]. Breuer et al. [14] investigate the structural, organisational and dynamic properties of actin networks in Arabidopsis in order to gain a quantitative understanding of the actin-based organelle transport. To be able to perform such an analysis, the authors develop a pipeline that can extract actin networks from confocal microscopic images of the actin cytoskeleton. In the segmentation stage of their pipeline, they implement a tubeness filter that uses multiscale Hessian matrix eigenvalues [74] before thresholding this filtered image with an adaptive median threshold. Subsequently, they skeletonise the binary image before continuing with the quantitative analysis. In another study focusing on the quantitative analysis of a plant cytoskeleton, Faulkner et al. [22] develop the software tool CellArchitect, which can detect drug-induced changes in the microtubule network organisation in Arabidopsis cells. Prior to microtubule segmentation, the authors identify and mask cell borders in the image, hence revealing each cell object. To segment the microtubule networks, they first apply a Gaussian filter for smoothing and denoising, and then binarise the filtered image via a percentile-based local threshold calculated separately for each cell. Binarised networks are then skeletonised before morphological quantification.

Other studies focusing on the morphological analysis of FtsZ scaffolds in Physcomitrella plastids, the plastoskeleton, apply deconvolution as a pre-processing step to 3D confocal microscopy images of FtsZ networks, followed by adaptive thresholding [6], [64]. The thresholded images are subjected to various global and local statistical analyses leading to extraction of a series of descriptive features. Another study by Asgharzadeh et al. [7] add modifications to the segmentation and quantification techniques introduced in Asgharzadeh et al. [6] and Özdemir et al. [64]. After segmentation and morphological analysis of the networks in 3D image form, the binary network images are transformed into volume meshes and their mechanical behaviour is analysed using theoretical material parameters and finite-element modelling. The relationships between the cytoskeletal network morphology and the mechanical behaviour of the 3D simulations are examined using supervised machine learning. Finally, a comparative evaluation of this analysis for two closely related FtsZ isoforms from Physcomitrella is presented [7].

To illustrate the “filter-and-threshold” strategies, we here present an exemplary segmentation procedure for 3D images, explaining the multiple steps of a semi-automated pipeline (Fig. 1). The input image to this procedure is a confocal microscopy z-stack image of a filamentous network assembly belonging to the FtsZ1-1 isoform, a protein that localises to the chloroplasts of the moss Physcomitrella (Fig. 1A). This image was acquired using the molecular biology and imaging protocols described in Asgharzadeh et al. [6], Özdemir et al. [64] and Asgharzadeh et al. [7]. At the beginning of the pipeline, the image’s grey values are rescaled to a range of 0 and 1. The image is first subjected to a pre-processing step (Fig. 1A, Step-1) consisting of the following chain of operations: i) Gaussian filtering, ii) H-minima transformation, iii) Hessian-based vesselness enhancement filter [23], [4], iv) morphological grey-closing, v) adaptive histogram equalisation (CLAHE), vi) median filtering. The output of this process is an image, which is denoised, smoothed and enhanced for the vessel-like features (Fig. 1B). The enhanced image is segmented using hysteresis thresholding (Fig. 1, Step 2). The resulting binary image contains the network scaffolds corresponding to a multitude of chloroplasts (Fig. 1C). To isolate the scaffold corresponding to a single chloroplast, we mask the binary network image with a chloroplast mask, which corresponds to the largest central chloroplast in the image (Fig. 1, Step-3) (the chloroplast mask is separately generated from the chlorophyll fluorescence and not shown due to space limits). The resulting binary network image (Fig. 1D) is subjected to a tubular enhancement procedure (Fig. 1, Step-4) based on the following consecutive operations: i) Euclidean distance transform, ii) Hessian-based line filter [74], iii) adaptive histogram equalisation (CLAHE), iv) local adaptive thresholding following Bernsen’s method [10], v) morphological binary-closing, vi) removal of binary noise with a size filter. Output of this procedure is a binary image that has been substantially thinned and has a uniform width (Fig. 1E). Therefore, this image is already suitable for the detection of interest points (nodes, endpoints, etc.). Optionally, the image can be skeletonised via morphological thinning algorithms (since the image at this stage has a uniform width, morphological thinning can be exerted on it without causing artefacts). In our example here, the image is skeletonised based on Lee et al. [43] (Fig. 1, Step-5). The resulting skeleton image (Fig. 1F) is further processed with an algorithm (Fig. 1, Step-6) that transforms the raw skeleton into an annotated network (Fig. 1G), specifically labelling the internal nodes (red), the endpoints (yellow) and the connecting edges (green) of the skeleton. The representation of a filament network in the form of such an annotated network offers a wide range of quantitative analysis options. For instance, geometrical features such as edge directionality, lengths, tortuosity, etc. can be directly computed from this representation. The annotated network can also be easily transformed into a morphological graph to implement graph-theoretical algorithms on it (for instance, to compute shortest paths between selected nodes). Finally, an overlay of the annotated network and the binary region mask is shown in Fig. 1H. Parameters used in the described implementation are given in Table 1.

Fig. 1.

Illustration of a typical filter-and-threshold procedure in a network extraction procedure applied to a scaffold of FtsZ1-1 filaments in the chloroplasts of Physcomitrella. (A) Input is a raw 3D image of FtsZ1-1 produced by confocal microscopy. (B) Enhanced image after step-1. Step-1 consists of the following chain of operations: i) Gaussian filtering, ii) H-minima transformation, iii) Hessian-based vesselness enhancement filter, iv) morphological grey-closing, v) adaptive histogram equalisation (CLAHE), vi) median filtering. (C) Binary image produced by Step-2, which segments the enhanced image using hysteresis thresholding. (D) Binary image produced by Step-3, which masks the image in (C) to isolate the FtsZ scaffold corresponding to a single chloroplast. (E) Binary tubular image after Step-4, which consists of the following chain of operations: i) Euclidean distance transform, ii) Hessian-based line filter, iii) adaptive histogram equalisation (CLAHE), iv) local adaptive thresholding, v) morphological binary-closing. vi) removal of binary noise with a size filter. (F) Binary skeleton image produced by Step-5, which applies a 3D skeletonisation algorithm to the tubular image in (E). (G) Feature image after Step-6, which finds the nodes, endpoints and edges of the skeleton image in (F). (H) Overlay of the images in (D) and (G). Intensity values in (A) and (B) are represented with the “viridis” colourmap, which has a spectrum from purple (lower values) to yellow (higher values). (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

Table 1.

Parameters used in different steps of the pipeline implemented in Fig. 1. Local windows and structuring elements are given with the order of dimensions (z, x, y). Note that the steps that need no manual parameter input are excluded from this table.

Process	Step	Parameters
Pre-processing	Gaussian filter	sigma = 1.0
	H-minima transform	h = 0.03
	Multiscale Hessian-based vesselness enhancement filter	sigma minimum = 0.8, sigma maximum = 5.0, sigma number = 9.0, alpha = 2.0, beta = 0.5, gamma = 0.005, objectness scaling = True
	Morphological grey-closing	structuring element = ellipsoid with diameters (3, 7, 7)
	CLAHE	window = (9, 9, 9), contrast limit = 0.1
	Median filtering	window = (1, 5, 5)
Binarisation	Hysteresis threshold	high threshold = Otsu threshold, low threshold = 0.75 × Otsu threshold
Skeletonisation	Hessian-based line filter	sigma = 1.5, alpha1 = 0.5, alpha2 = 2.0
	CLAHE	window = (7, 7, 7), contrast limit = 0.05
	Local adaptive thresholding	window = (5, 5, 5), contrast threshold = 0.1
	Morphological binary-closing	structuring element = ellipsoid with diameters (3, 7, 7)
	Size filtering	size threshold = 20 voxels

4.2. Limitations of conventional segmentation methods for cytoskeleton images

The threshold-based segmentation methods such as exemplified above are still popular due to their simplicity, especially when the aim of the segmentation task is simply to obtain a binary mask for the foreground voxels corresponding to the object. When the sample images already have a high signal-to-noise ratio, the preliminary denoising/feature-enhancement filters can sufficiently improve the image so that a simple thresholding can yield a decent representative binary mask. With more challenging images, for instance, images with low signal-to-noise ratio, or images with discontinuous or blurred contours, more complex strategies are required for segmentation. In addition to these general issues, a particular problem related to cytoskeleton imaging is that the physical width of the filaments is usually smaller than the resolution limit of many microscope modalities (e.g., confocal laser scanning microscopy) [88], a limitation that obscures the correct filament tracks in the image. In connection with this, multiple adjacent filaments are unresolvable within the diffraction limit of light. In such cases, quantitative measurements are jeopardised by the lack of precision in localisation of the filament tracks in the image. The aforementioned threshold-based methods are incapable of handling the problem of imprecise localisation, since they usually yield a pixel/voxel mask that is wider than the physical width of the polymer. Furthermore, these pixel/voxel masks are not convenient for direct quantitative analysis and usually subjected to a morphological thinning operation (skeletonisation) [14], [87]. Morphological thinning, however, does not guarantee accurate localisation of the physical filaments and, depending on the degree of local variations in object thickness, tends to generate artefacts such as incorrect nodes and endpoints. A more accurate localisation and extraction of the filaments in the images involves detection of local maxima (ridges) along the length of the filaments (a process referred to as tracing). Multiple approaches are being used to tackle the tracing task. Rigort et al. [67] develop a method to extract actin filaments from cryo-electron tomograms based on generic template matching. Template matching yields a local cross correlation map and an orientation field, both of which are used to develop a tracing algorithm that extracts the filament centrelines. An advantage of this method is that the user has the flexibility to modify the cylindrical template in order to detect different cytoskeletal filament types such as intermediate filaments and microtubules. This segmentation strategy was also employed by Jasnin & Crevenna [31]. Rogge et al. [68] propose a method for segmentation and quantification of F-actin filaments from 2D fluorescence microscopy images. Their segmentation relies on an iterative process consisting of a pre-processing step (involving Gaussian filter, morphological operations and thresholding), a tracing step for centreline extraction and a step for fibre connection. With each consecutive cycle of the process, filaments with larger width are extracted. The segmented images are then subjected to quantitative analysis. They also provide an open-source GUI, FSegment, which performs the presented method. Costigliola et al. [19] use a method (originally proposed by Gan et al. [25] that combines multi-scale steerable filters and a non-maximum suppression operator for direct extraction of vimentin network skeletons from spinning disk confocal microscopy (SDCM) images. The multiple skeleton segments are merged into a full network based on their proximity and orientations. In the second step of the analysis pipeline, the authors apply a length threshold to distinguish the longer, more bundled vimentin filaments from the shorter fragments. Finally, they subject the resulting vimentin networks to various quantitative analyses, which include an analysis of the filament orientation in relation to the direction of cell movement. Tsugawa et al. [82] develop a method for extraction of local anisotropy vectors and orientation of fibrillary structures in 2D images based on nematic tensor analysis (NTA). After the development and validation of their method using synthetic images, they demonstrate its performance with real data by applying it to real confocal microscopy images of cortical microtubules in a giant cell of an Arabidopsis sepal.

4.3. Model-based approaches

To attain a greater accuracy in localisation of the filament ridges in the images, model-based strategies using various optimisation algorithms are often employed in the segmentation/tracing tasks. In the specific case of cytoskeleton image segmentation, the objective of most model-based methods is to fit a smooth curve to the centrelines of the filaments with as high precision as possible [78], [70], [84], [91], [88].

To this end, a group of methods, namely deformable models, are popular. These methods are based on deformable contours (curves for 2D images or surfaces for 3D images) that are defined within the sample image frame, and that undergo gradual deformation under the influence of internal and external forces. This so-called “contour evolution” involves iterative modification of the contour so that it approximates desired image features, such as object boundaries (or ridges in the specific case of filaments) while being constrained by certain boundary conditions. The internal forces are defined within the contour, and are responsible for constraining the evolving contour according to prior knowledge about the object shape (e.g., object’s local smoothness or curvature). The external forces, on the other hand, are calculated from the image and specify the directions and speeds that are used to drive the contour. The ability to design the internal and external forces gives the user control over the model, which can thus be tuned to segment, for example, images with different modalities.

Since the seminal paper by Kass et al. [35], a large body of research was conducted on deformable models, and a wide range of segmentation techniques were developed. For a deep introduction into the field, the reader can refer to Tsechpenakis [81] and Jayadevappa et al. [32]. In the following, we will introduce several recent studies that use deformable model-based strategies to segment/trace cytoskeletal filaments in microscopic images.

Regarding the application of deformable models in the frame of segmentation and tracking of cytoskeletal networks, a particular subcategory of deformable models, namely parametric deformable models (active contours or snakes), is often preferred. In one of the early examples of these studies, Kong et al. [39] employ particle filters to track the microtubule tip, and then, based on the identified tip locations, they segment the microtubule filaments via open active contours. In another study focusing on microtubule segmentation, Nurgaliev et al. [62] combine active contours with Monte Carlo simulations to identify microtubule trajectories in 3D electron tomograms.

In a series of original publications [45], [44], [46], the authors introduce “stretching open active contours” (SOACs), which have since developed into popular tools for the extraction and quantification of cytoskeletal networks. Differing from the typical active contour models, which use closed curves that evolve to find the object boundaries, SOACs are open curves, which start stretching from snake tips and eventually delineate the central lines (ridges) of the filaments in the image. Smith et al. [78] develop a user-interactive software based on SOACs, named JFilament, which does not only segment and track filaments in microscopic images but can also be used to extract certain static and dynamic quantitative features of the filaments. Xu et al. [90] develop a method that achieves automated, simultaneous initialisation and evolution of multiple SOACs. This method automatically extracts the centrelines of an entire filament network. Furthermore, this method uses graph partitioning strategy to reorganise the evolved SOACs in order to dissect and label individual filaments in the network. Xu et al. [91] improve the SOACs-based segmentation algorithms and extend them to 3D images. In particular, they introduce an adaptive stretching force that results in robust contour evolution under high intensity variations and noise. Xu et al. [92] develop the open-source user-interactive platform SOAX that uses SOACs algorithms to extract cytoskeletal networks from images, and additionally provides an option to perform quantitative analysis on the segmented objects based on a set of input parameters from the user. Xu et al. [89] adds temporal dimension to the SOAX software. The final version of the software (TSOAX) is capable of tracking and analysing cytoskeletal networks in time-series movies consisting of both 2D and 3D images. Kotsur et al. [40] address some drawbacks of SOACs in tracking of individual intermediate filaments in confocal time-series images and propose a modified method that solves these issues. Their method can accurately track individual filaments within their branched network due to a reconfigured active-contour algorithm that has a better control on the snake endpoint growth.

Another user-interactive application, designed to segment and analyse filamentous and fibrous objects in microscopic images, is proposed by Usov & Mezzenga [83]. Their tool FiberApp works on images acquired by any type of microscope, although the presented results are mostly based on atomic force microscopy (AFM) or electron microscopy (EM) images. They use a combination of the A* pathfinding algorithm and the active contour models to extract the contours of fibrous objects from the images (the “contour” here referring to the fibre centrelines). FiberApp enables an option for the user to specify heterogeneous stiffness values for different regions of the image. This can be used for instance, to remove undesirable fluctuations in the extracted contours. It also offers a range of options for quantitative analysis, which can be automatically applied to the traced fibres.

Further model-based strategies include Valdman et al. [84] who focus on devising methods for inferring material properties of cytoskeletal filaments from exhibited filament shapes, using simulated, noisy biopolymer images with known stiffness. They test their methods also on real Taxol-stabilised microscopic images of microtubules. Their approach relies on optimising an objective function that quantifies the overlap between an open smooth curve designed in the orthogonal polynomial basis and the fluorescence intensity in the image. Xiao et al. [88] combine curve-fitting and level-set strategies for the centreline extraction from filament images. By representing the open curves with two B-spline vector level sets, they can formulate filament centreline extraction as a global convex optimisation problem. Their method achieves sub-pixel accuracy without prior knowledge of the filament number in the image. Similar approaches are applied to the analysis of cytoskeletal dynamics as well. In such a study, Kapoor et al. [34] develop the multi-purpose software MTrack for microtubule detection, tracking and analysis in time-series images acquired via TIRF microscopy. They use Maximally Stable Extremal Regions (MSER) algorithm to obtain restricted image areas containing the microtubule seeds. Subsequently, the seed endpoints, from which the microtubules will grow and shrink, are detected by fitting a Sum of Gaussians (SoG) model to each of the detected seed regions. This cycle of implementing MSER and fitting a SoG model is repeated for consecutive time frames to track the filament endpoints (hence the growth/shrinking of filaments) over time, using the information from a successfully segmented time frame as starting point for the model-fitting in the next time frame. By representing the SoG path with a 3rd order polynomial function, the authors enable their method to robustly track straight, curved and crossing filaments. MTrack also offers an analysis option for microtubule length over time.

Another popular tool for filament tracing and tracking is proposed by Ruhnow et al. [70] who introduce the semi-automatic software application FIESTA, which is capable of centreline and tip extraction from filament images with sub-pixel precision as well as their tracking in time-series images. FIESTA relies on an initial binarisation, thinning and region-of-interest definition, which is followed by fitting a 2D model based on Gaussian distributions to identify the centre positions of filaments and tips. Finally, a spline interpolation joins the segments together to obtain the centrelines. After the centreline extraction from each timeframe in a time-series, the algorithm establishes the temporal connections between the detected objects to complete the tracking. Other related studies focusing on filament tip tracking include Hadjidemetriou et al. [27], Demchouk et al. [20], Prahl et al. [66], Maurer et al. [56], [12]. Importantly, the methods described in these studies aim to identify and track microtubule tips, rather than tracking the end-binding proteins.

Breuer and Nikoloski [13] introduce an open-source software for automatic decomposition of a network into its constituent filaments. They treat filamentous networks as weighted geometric graphs, where the edges correspond to segments of the filaments and the nodes correspond to the segment endpoints. As an objective of the optimisation process, they define a “roughness” term that represents the variability in filament thickness. Their proposed method seeks to assign segments to paths of filaments while minimising the total roughness as the entire network is covered. In addition to the total roughness, their method can also implement alternative optimisation objectives such as average roughness or curvature-related measures. The authors transform their method into a software tool, DeFiNe, and test it on a series of image-based network data, including LSCM images of the Arabidopsis actin cytoskeleton.

Park [65] proposes a method for segmentation, skeletonisation and quantitative geometrical analysis of 3D filament networks from LSCM images. For segmentation, this author combines structure tensor eigenvalues and Otsu threshold calculation to derive an energy functional, which is minimised using the graph-cuts method. After binarisation, a skeleton-based seed selection is applied, followed by multiple hypothesis template tracking [24] for accurate centreline extraction. Finally, for a quantitative analysis, Park [65] calculates a distribution of the fibre diameters over their reconstructed fibrillary network.

Model-based approaches are also used for the segmentation of cytoskeletal filaments from electron microscopy images. Kervrann et al. [36] propose a probabilistic model in the frame of conditional random fields (CRFs) for segmentation of microtubules from 2D sections of cryo-electron tomograms. In another study focusing on extraction of filaments from cryo-tomograms, Sazzed et al. [75] introduce the software tool BundleTrac for semi-automatic tracing of individual actin filaments in filament bundles in noisy cryo-tomograms. BundleTrac first detects the main axis of the bundle and applies longitudinal averaging along this axis for denoising. For tracing of the filaments, BundleTrac optimises a 2D seven-peak Gaussian convolution. Yue et al. [93] propose a segmentation method for cryo-EM images of microtubules. This method denoises and enhances the cryo-EM image through extensive pre-processing, including an improved diffusion filter, and then applies an adapted Chan-Vese [15] algorithm for segmentation of the filtered image.

We here demonstrate the model-based approaches with two illustrative implementations (Fig. 2), where two different deformable model-based strategies are applied to a confocal microscopy image of the Physcomitrella FtsZ1-1 isoform. The first method is binary region segmentation using a technique called Morphological Active Contours without Edges (morphological ACWE), which is an adaptation of the Chan-Vese method by Márquez-Neila et al. [52]. We demonstrate this method by using the open-source python code provided by the authors. The second method is filament centreline extraction by using the SOAC method, which we implement by using the JFilament software [78]. For the demonstration of these methods, we use the same input image as the one in Fig. 1 (for the details about the molecular biology and the image acquisition, we refer the reader to Asgharzadeh et al. [6], Özdemir et al. [64] and Asgharzadeh et al. [7]). Fig. 2A outlines the implementation of the morphological ACWE on the 3D image. The pre-processing that transforms the raw image into a state with reduced noise and enhanced tubular structures comprises the same sequence of steps as described in Fig. 1B. Subsequently, morphological ACWE is performed on this enhanced image. The iterative segmentation process is initialised with a binary checkerboard pattern (Fig. 2A, iteration 0). The contour evolves under external and internal forces, eventually approximating the shape of the objects in the image. The different states of the contour evolution are demonstrated (Fig. 2A, Iteration 10, 15 and 20). It is important to note that in this case the pre-processing is essential for the segmentation to be successful, whereas the method yields a poor segmentation if the raw image is chosen as the initial input. The end product of this pipeline is a region mask, which can be thinned and skeletonised to perform a quantitative network analysis (as in Step-4 to Step-6 in Fig. 1). An alternative to the approach described in Fig. 2A would be a direct centreline extraction by curve-fitting to the raw data. This would more accurately predict the true filament ridges. Fig. 2B shows such an implementation of the centreline extraction from the raw image using the JFilament software [78]. For this example, we select a 2D slice from our image (Fig. 2B, left), since the strong anisotropy in the resolution of our 3D image hinders a successful implementation. In this semi-automated process, we select the initial positions of the snakes and let them deform so that they delineate the ridges of the filaments. The traced centrelines are shown in Fig. 2B (middle) and their overlay with the raw image is shown in Fig. 2B (right). The parameters that we use for the implementations in Fig. 2A and Fig. 2B (except for the pre-processing of the input image in Fig. 2A, which is the same as in Fig. 1) are given in Table 2.

Fig. 2.

Illustration of model-based approaches to a scaffold of Physcomitrella FtsZ1-1 filaments. (A) Implementation of the morphological ACWE method for binary region segmentation. Raw image is first subjected to a pre-processing step leading to a denoised and vessel-enhanced form of the image. The contour is initialised as a checkerboard level set (iteration 0), which evolves under internal and external forces. The different states of the evolving contour are shown (Iterations 10, 15 and 20). (B) Implementation of the SOAC method using the JFilament software. The input is a 2D slice (left plane) from the 3D raw image. Snakes are initialised manually and then allowed to deform to delineate the ridges of the filaments (middle plane). An overlay of the fitted centrelines and the raw image is shown in the right plane.

Table 2.

Parameters used in the two methods implemented in Fig. 2.

Process	Parameters
Morphological ACWE	smoothing = 1.0, lambda1 = 1.0, lambda2 = 1.6
SOAC	stretch = 100.0, spacing = 5.0, background = 10.0, alpha = 5.0, weight = 5.0, smoothing = 1.5, foreground = 64.0, beta = 0.2, gamma = 10000.0

5. Deep learning methods for enhancement and segmentation of cytoskeletons

5.1. Deep learning for segmentation

The recent advances in the field of deep learning (DL from now on) and its implementations in computer vision had a great impact on bioimage segmentation, resulting in the development of an array of neural networks specialised for segmentation tasks [69], [18]. The accumulation of microscopic images of the cytoskeleton along with the invention of various automated and semi-automated software tools for filament segmentation/tracing leads to faster and more accurate cytoskeleton image annotation, and thus the accumulation of ground-truth image data for supervised learning studies. This in turn leads to the development of DL-based segmentation methods tailored to cytoskeleton images. In addition to enhancing the segmentation accuracy, these methods also aim to find solutions to the problems that are frequently encountered in filament segmentation but cannot be robustly overcome using the unsupervised segmentation methods discussed above.

Asgharzadeh et al. [8] demonstrate an application of U-net for the segmentation of FtsZ networks in Physcomitrella chloroplasts, the plastoskeleton, from confocal microscopy images. In another study based on U-net, Liu et al. [48] develop a segmentation method for cytoskeleton images (actin filaments and microtubules) acquired using confocal microscopy. To generate the ground-truth images, they rely on a combination of i) SOAX-based segmentation, ii) a single U-net module fit to the SOAX output and iii) manual correction of the segmentation by this U-net module. Their proposed network is based on multiple modified U-net modules, which are stacked in an end-to-end manner. This network, when trained on the ground-truth, performs slightly better than a single U-net module on confocal microscopy images of actin and microtubule networks.

Obtaining high-quality ground-truth images is a general challenge for most projects using DL-assisted segmentation. To reduce the labour associated with image annotation, weakly supervised approaches are often used. In a recent example of such studies, Lavoie-Cardinal et al. [41] use a modified U-net, trained on weak annotations in the form of polygonal bounding boxes, for segmentation of F-actin nanostructures from STED images. Bilodeau et al. [11] take a further step and introduce MICRA-Net, a neural network designed to be trained on image-level classification annotations in order to perform multiple microscopy tasks, including semantic segmentation of images. In addition to other cell biological use cases, the authors demonstrate efficient segmentation of F-actin nanostructures from STED images using a model trained on image-level annotations.

One of the tricky tasks relating to the segmentation of cytoskeletal networks is the correct identification of filaments at the intersection points of the networks. This issue is especially complicated with dense networks and bending filaments. Liu et al. [47] tackle this challenge by proposing a U-net-based neural network, which accepts binary network images as inputs, and dissects the network into individual filament instances. Since the network is trained on orientation-associated ground-truth, it does not confuse the filament identity at the intersection points. The downside of this approach is that any filaments that are kinked/curved will be fragmented into different orientation groups. The method also handles this issue by using an algorithm that repairs these fragmented filaments based on the orientation vectors at the filament termini. The authors test their method on microtubule images, which they first segment using the method by Liu et al. [48] to obtain the binary input data.

Liu et al. [49] propose a DL-based method for geometrical and topological characterisation of actin filament networks. For an initial binary segmentation, they use the U-net based method by Liu et al. [48]. For the topological characterisation, they use the ResNet platform to train a network that accepts the binary images as inputs and generates heatmaps to highlight the junctions and endpoints of the networks. For the quantification of the filament lengths, they employ a fast-marching algorithm, which calculates a geodesic distance map by using the key points (junctions and endpoints) as the seeds. The number and lengths of the filaments can then be acquired from the local peak values obtained from this distance map.

In another recent study, Eckstein et al. [21] propose a microtubule tracking method for large electron microscopy volumes. Their workflow, implemented on Drosophila neural tissue, starts with training of a 3D U-net (based on ground-truth skeleton annotations) that predicts scores for each voxel’s belonging to a microtubule. From the resulting probability map, they extract candidate points for microtubule segments through non-maximum suppression and thresholding. These candidates are then translated into a graph representation, where the nodes of the graph represent the candidates and the edges show potential links between them. Finally, they formulate the identification of the true microtubule track as a constraint optimisation problem, which aims to find the most appropriate subset of edges in the graph. Solving this optimisation reveals the correct microtubule tracks.

DL-based strategies are also beginning to be applied to filament tracking in time-series data. Masoudi et al. [54] develop a method for tracking of microtubules in 2D time-series images and measuring the velocities of microtubules. They tailor their method particularly to combat the challenges of the TIRF imaging method. Their method relies on instance-level segmentation of microtubules at each frame of the time course and then generating a trajectory for each microtubule instance over time. Prior to the segmentation, their method uses a visual attention module, which contains a CNN and a Recurrent Neural Network (RNN) that processes the image and suggests where to focus for segmentation. Then the segmentation module, which is an encoder-decoder system, segments the image within this attention region. After the segmentation is finished, the individual microtubule instances are associated by analysing successive pairs of time frames in order to get the trajectory for each microtubule.

5.2. Deep learning for enhancement

In addition to boosting the general image processing tasks such as segmentation and tracking in a broad sense, DL contributes to the handling of microscopy-specific problems. Enhancement of microscopic resolution is a topic, which attracts much interest in recent years and benefits greatly from DL-based technologies. One important contribution to this field was made by Ouyang et al. [63], who introduce ANNA-PALM, a DL strategy that generates super-resolution images from sparse localisation-microscopy images and/or widefield images. Localisation microscopy techniques include PALM/STORM and DNA-PAINT, both of which are used by Ouyang et al. [63] to acquire microtubule images. The ground truth for the learning task is composed of dense, super-resolution PALM images. To build the training image set, under-sampled, sparse versions of the same PALM images (and optionally a widefield image) are used. The training is performed via pix2pix image-to-image translation architecture [30] and aims to enable the neural network to reconstruct a super-resolution image from a sparse localisation image (or even a widefield image if included in the training). The authors demonstrate that their method performs well on a variety of image datasets representing different subcellular structures, including microtubules. As a result, the method is a suitable option for a tubeness-enhancement step prior to filament segmentation/tracing studies.

Indeed, Nanguneri et al. [60] employ this approach as they utilise ANNA-PALM as a part of their computational analysis framework focusing on F-actin network in dendritic spines. They develop a pipeline for segmentation and quantification of actin filament networks in spine-specific regions of neurons using super-resolution image datasets. They first use a supervised machine-learning tool, named trainable Weka segmentation [5], to identify actin-rich regions in the images. To achieve fine segmentation of the actin filaments within these regions, they compute a tubular model of their super-resolution images using ANNA-PALM [63], in particular using the microtubule models offered by that study. Subsequently, they apply certain masks on the tubular model to obtain region-of-interests, which are actin-rich areas spatially correlating with a postsynaptic marker. Within these specific regions, they finally apply a ridge detection algorithm [79] to extract actin skeletons, which are then subjected to quantifications. Concurrent with the analysis of the actin cytoskeleton, supervised learning algorithms are used to perform a morphological characterisation of the spines.

Lee et al. [42] propose another DL-based technique for tubular image enhancement. They train a CycleGAN [97] model that is capable of enhancing the quality of low-resolution microtubule network images, which then can be effectively segmented simply using Otsu thresholding. They train their model on two sets of images. One set, acquired from the Human Protein Atlas [80], consists of high-resolution confocal images, which represent the reference images for training. The second set, acquired from the Broad Bioimage Benchmark Collection [50] are low-resolution widefield images representing targets for enhancement. The CycleGAN model being trained on these two sets learns the data distributions in both the low and high-resolution image sets and can eventually transform an input image from a low-resolution distribution to an output with a high-resolution distribution.

A study combining the two technologies of super-resolution imaging and DL is conducted by Jin et al. [33], who propose deep-learning-assisted structured illumination microscopy (DL-SIM). This method uses DL to boost the performance of SIM by enhancing the image reconstruction step so that the image reconstruction can be performed with only few raw images and using images acquired under low-light conditions. Using the U-net architecture, the authors train two networks on raw SIM images of microtubule and actin networks, in addition to other subcellular structures. For training of the first network, they use raw SIM data as input and the standard SIM reconstruction images as the ground truth. This network can achieve successful reconstruction with drastically reduced numbers of raw images compared to the conventional SIM reconstruction methods. The second network, consisting of two U-nets, is trained to handle noisy images. For this network, the images acquired under low-light conditions are used as training input, whereas the corresponding SIM reconstructions from images acquired under normal-light conditions are used as the ground truth. The resulting network is capable of SIM reconstruction from raw images acquired under extremely low-light conditions. Chaining of the two U-nets yields a tool that can reconstruct SIM images both using fewer raw images than needed by the conventional SIM reconstruction methods and achieves better reconstruction quality with noisy raw data. Therefore, the method potentially reduces the photobleaching problems associated with prolonged imaging and strong illumination.

6. Summary and outlook

High-resolution imaging combined with the advanced computational methods is boosting the information acquired from bioimage analysis. Enhancement, segmentation, tracing and tracking methods are being developed for cytoskeleton images with several user-interactive software tools introduced already, as covered in this review. Table 3 summarises the reviewed works, linking each study with the respective bioimaging method, the biopolymer type, the segmentation/enhancement tasks, key strategies employed and, if available, the name of the user-interactive tool.

Table 3.

A summary of the publications covered in this review.

Category	Publication	Bioimaging Technique	Biopolymer Type	Main Segmentation/Tracing/Enhancement Tasks	Key Strategies	Relevant User-Interactive Tool
Intensity threshold	[36]	Cryo-Electron Tomography	Microtubules	Region segmentation (pixel/voxel mask)	Conditional Random Fields-Maximum a Posteriori estimation	None
	[3]	Fluorescence Microscopy	Actin filaments	Region segmentation (pixel/voxel mask)	Image decomposition, multiscale line filters, adaptive thresholding	None
	[94]	SMLM	Microtubules	Region segmentation (pixel/voxel mask)	LFT, OFT	SIFNE (SMLM Image Filament Network Extractor)
	[14]	Confocal Microscopy	Actin filaments	Region segmentation (pixel/voxel mask)	Multiscale Hessian-based tubeness filter, adaptive thresholding	None
	[22]	High Throughput Confocal Microscopy	Microtubules	Region segmentation (pixel/voxel mask)	Gaussian filter, local thresholding	CellArchitect
	[6]	Confocal Microscopy	FtsZ	Region segmentation (pixel/voxel mask)	Deconvolution, local thresholding	None
	[64]	Confocal Microscopy	FtsZ	Region segmentation (pixel/voxel mask)	Deconvolution, local thresholding	None
	[87]	STORM	Actin filaments	Region segmentation (pixel/voxel mask)	LFT, OFT, H-minima transform, Meyer Watershed transform	None
	[7]	Confocal Microscopy	FtsZ	Region segmentation (pixel/voxel mask)	Deconvolution, local thresholding	None
Filter/template-based tracing	[67]	Cryo-Electron Tomography	Actin filaments	Centreline extraction	Template matching,tracing	Actin Segmentation (an AMIRA extension package)
	[31]	Cryo-Electron Tomography	Actin filaments	Centreline extraction	Template matching, tracing	None
	[25]	SDCM/SIM	Microtubules and Intermediate Filaments	Centreline extraction	Steerable filters	None
	[82]	Confocal Microscopy	Microtubules	Estimation of local anisotropy and orientation	Nematic tensor analysis	None
	[19]	SDCM	Intermediate Filaments	Centreline extraction	Steerable filters	None
	[68]	Widefield Fluorescence Microscopy	Actin filaments	Centreline extraction	Filtering and tracing	FSegment
Conventional model-based	[39]	SDCM	Microtubules	Tracking over time	Open active contour model	None
	[27]	Epifluorescence Microscopy, Confocal Microscopy	Microtubules	Centreline extraction, tip tracking over time	Optimisation using consecutive level-sets method	None
	[45]	TIRFM	Actin filaments	Centreline extraction,tracking over time	Open active contour model	None
	[44]	TIRFM	Actin filaments	Centreline extraction,tracking over time	Open active contour model	None
	[46]	TIRFM	Actin filaments	Centreline extraction,tracking over time	Open active contour model	None
	[62]	Cryo-Electron Tomography	Microtubules	Centreline extraction	Open active contour model	None
	[78]	TIRFM, SDCM	Actin filaments	Centreline extraction,tracking over time	Open active contour model	JFilament (an ImageJ plugin)
	[20]	Digital Fluorescence Microscopy	Microtubules	Tip tracking over time	Gaussian fitting, Gaussian survival functions	None
	[70]	TIRFM, Epifluorescence Microscopy	Microtubules, Kinesin motors	Centreline extraction,tracking over time	Least squares withGaussian distribution models	FIESTA
	[90]	TIRFM, SDCM	Actin filaments	Centreline extraction	Open active contour model	None
	[84]	TIRFM	Microtubules	Centreline extraction	Active contour model	None
	[56]	TIRFM	Microtubules	Tip tracking over time	2D model fitting, modifications to Ruhnow et al. [70]	None
	[66]	Epifluorescence Microscopy	Microtubules	Tip tracking over time	Gaussian fitting, Gaussian survival functions	TipTracker
	[91]	TIRFM, SDCM	Actin filaments	Centreline extraction	Open active contour model	None
	[12]	TIRFM	Microtubules	Tip tracking over time	2D model fitting, modifications to Ruhnow et al. [70]	None
	[13]	Confocal Microscopy	Actin filaments	Decomposition of network into constituent filaments	Graph partitioning optimisation	DeFiNe
	[92]	TIRFM, SDCM	Microtubules, Actin filaments	Centreline extraction	Open active contour model	SOAX
	[83]	AFM, EM	Fibrils of nanocellulose, BSA, polysaccharides,amyloid, beta-lactoglobulin	Centreline extraction	A* Pathfinding Algorithm, curve fitting	FiberApp
	[88]	Fluorescence Microscopy, Phase-Contrast Microscopy,Darkfield Microscopy	Microtubules, Axonomes	Centreline extraction	B-spline vector level sets, generalised linear model	None
	[93]	Cryo-Electron Microscopy	Microtubules	Region segmentation (pixel/voxel mask)	Chan-Vese model	None
	[65]	Confocal Microscopy	Not specified	Region segmentation, fiber reconstruction	Graph-cuts, template fitting, multiple hypothesis tracking	None
	[75]	Cryo-Electron Tomography	Actin filaments	Centreline extraction	2D convolutional optimisation using Gaussian kernels	BundleTrac
	[89]	TIRFM, SDCM	Actin filaments, myosin rings, fibrin bundles	Centreline extraction,network tracking over time	Open active contour model	TSOAX
	[34]	TIRFM	Microtubules	Centreline extraction,tracking over time	Sum of Gaussian (SoG) and polynomial models	MTrack (an ImageJ plugin)
	[40]	Confocal Microscopy	Intermediate Filaments	Centreline extraction and tracking over time	Open active contour model	None
DL-assisted model-based	[63]	PALM, DNA-PAINT, Widefield Microscopy	Microtubules	Superresolution reconstruction	U-net and GANs	ANNA-PALM (ImageJ plugin and web application)
	[42]	Confocal Microscopy, Widefield Microscopy	Microtubules	Resolution enhancement	Cycle-GAN	None
	[47]	Not specified	Microtubules	Instance segmentation of filaments of a network	Deep learning model based on U-net architecture	None
	[48]	Confocal Microscopy	Microtubules, Actin filaments	Centreline extraction	Deep learning model based on U-net architecture	None
	[60]	dSTORM	Actin filaments	Region segmentation, centreline extraction	ANNA-PALM, supervised learning, Gaussian derivatives	None
	[8]	Confocal Microscopy	FtsZ	Region segmentation (pixel/voxel mask)	Deep learning model based on U-net architecture	None
	[11]	STED	Actin filaments	Region segmentation (pixel/voxel mask)	CNN, weak supervision,latent learning	None
	[21]	Electron Microscopy	Microtubules	Reconstruction of filament tracks	3D U-net, non-maximum suppression, integer linear programming	None
	[33]	SIM, TIRFM	Microtubules, Actin filaments	Superresolution reconstruction	U-net	None
	[41]	STED	Actin filaments	Region segmentation (pixel/voxel mask)	U-net, weak supervision	None
	[49]	Not specified	Actin filaments	Junction and endpoint detection in a network	Deep learning model based on ResNet architecture	None
	[54]	TIRFM	Microtubules	Instance segmentation, time-tracking of filaments	Convolutional Neural Networks, Recurrent Neural Networks	None

These methods and tools enable morphological characterisation of cytoskeletal filaments and networks, which in turn reveal insights into their functional characteristics, such as efficiency of transport and adaptive mechanical behaviour. At the current stage, many existing methods and tools are still semi-automated, requiring extensive manual parameter configuration. Table 3 shows that there is an increasing trend towards using DL-assisted methods to automate various tasks in microscopic image analysis. One of the main limitations of using supervised DL methods in microscopic image processing/analysis is the difficulty and cost of data annotation. With large amounts of 3D/4D image data being now routinely produced in life sciences, this problem becomes even more restrictive. It is, therefore, reasonable to expect that more research will be focusing on unsupervised as well as semi- and weakly supervised learning strategies to tackle this problem. Many of the existing methods are still restricted to 2D static images. Considering the importance of 3D cytoskeletal dynamics for many cellular processes, future research will probably address robust tracking of individual filaments in cytoskeletal networks in 3D time-series images from the viewpoint of instance segmentation.

With the fast accumulation of image data, availability of advanced DL algorithms and GPU-accelerated hardware, further progress is likely to be achieved in the field, especially on the aspects of automation and 3D/4D extension.

CRediT authorship contribution statement

Bugra Özdemir: Investigation, Software, Writing - original draft. Ralf Reski: Supervision, Funding acquisition, Project administration.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.