Removal of Vesicle Structures From Transmission Electron Microscope Images

Abstract

In this paper, we address the problem of imaging membrane proteins for single-particle cryo-electron microscopy reconstruction of the isolated protein structure. More precisely, we propose a method for learning and removing the interfering vesicle signals from the micrograph, prior to reconstruction. In our approach, we estimate the subspace of the vesicle structures and project the micrographs onto the orthogonal complement of this subspace. We construct a 2d statistical model of the vesicle structure, based on higher order singular value decomposition (HOSVD), by considering the structural symmetries of the vesicles in the polar coordinate plane. We then propose to lift the HOSVD model to a novel hierarchical model by summarizing the multidimensional HOSVD coefficients by their principal components. Along with the model, a solid vesicle normalization scheme and model selection criterion are proposed to make a compact and general model. The results show that the vesicle structures are accurately separated from the background by the HOSVD model that is also able to adapt to the asymmetries of the vesicles. This is a promising result and suggests even wider applicability of the proposed approach in learning and removal of statistical structures.

I. Introduction

In structural biology, a long term target has been to image membrane proteins in their natural environment and to understand the functional organisation of the protein complex by reconstructing its 3d structure. In certain cases, the imaging of membrane proteins has been achieved by electron crystallography [33], however, many proteins are difficult to crystallize. More recently, the method of single-particle reconstruction (SPR) [12] has grown powerful enough to even reconstruct atomic structures. In SPR, cryo-electron microscopic projection images of similar copies of the protein are used for tomographic 3d reconstruction of the protein structure. A drawback of this method is the fact that viewing angles of the projection images are unknown that, together with a very low signal-to-noise ratio (SNR), makes the reconstruction problem particularly challenging.

In order to reconstruct membrane proteins, one additionally needs to separate the signal of the protein from the overlapping membrane vesicle. In [41] and [42], a new method, the random spherically constrained single-particle reconstruction, was introduced. There, the membrane proteins were reconstituted into lipid vesicles, frozen in vitreous ice and imaged by the microscope. This setup is illustrated in Fig. 1 and a sample micrograph with vesicles illustrated in Fig. 2. Membrane proteins were identified on the vesicle surfaces and the associated vesicle structures were removed using a scalable, geometric model, prior to the reconstruction. The protein reconstruction is however affected by the residual of the membrane vesicle structures near the proteins. This degrades the reconstruction quality, since the vesicle signal is much stronger than that of the protein, that even weak artifacts have a strong influence on the protein structure estimate. The reconstruction is additionally affected by other types of structural noise such as fragments of ice and random background noise.

Fig. 1.

Membrane protein reconstituted into a lipid vesicle, frozen in vitreous ice to be imaged in a cryogenic transmission electron microscope.

Fig. 2.

(a) Micrograph with vesicles where some of them contain a membrane protein particle (blue rectangles), one is amplified; (b) after the vesicle signal removal. A vesicle model must capture the variation in size and shape, yet separate structure from background noise and structural noise like pieces of ice and protein particles.

To extract the membrane proteins from the micrographs, we propose a unique approach where we consider the problem of separating three overlapping signals: 1) the vesicle, 2) the protein and 3) the background noise. The vesicle signal is dominant over those of the protein though there are strong intensity variations over all the micrographs that complicates the vesicle detection. The protein signal is weak and close to the noise level, and thus challenging to detect. Additionally, one has to automatically determine how complex the structures are. To solve the problems above, we must thus consider the three general problems addressed in the literature: signal separation, noise removal, and model selection.

In general, signal separation, aiming at recovering an unob-served signal from observed mixtures, has been widely studied in the literature. In blind signal separation, one assumes that one only observes mixtures of signals while the underlying components are unknown apart from the assumption that they are, for instance, minimally correlated, or maximally independent. Blind signal separation can be achieved for instance by principal component analysis (PCA) [19] its variants [6], [44], singular value decomposition (SVD) [39], independent component analysis (ICA) [16], independent subspace analysis (ISA) [17], multilinear PCA (MPCA) or higher order SVD (HOSVD) [24], [26]. Another approach is to impose structural constraints on the source signals, such as non-negativity imposing sparsity in the non-negative matrix factorisation (NMF) [7] and variance-difference maximising separation in common spatial pattern (CSP) [25]. If one of the signals to be separated is noise, the separation can be used as noise reduction technique, where the observed signal is decomposed to signal and noise subspaces and projected onto the signal subspace to remove the noise.

There is also a vast amount of other noise reduction techniques in the literature. The most primitive approaches are based on image filtering by simple linear or non-linear filters, such as local averaging or median filtering [31], [32]. These kinds of approaches, however, also obscure fine, low contrast details in the underlying image. The classic statistical approach for noise reduction is the Wiener filter [5], which is the optimal linear filter under the assumption of Gaussian noise. Edge preserving classic choices for noise reduction include total variation regularisation [36] and anisotropic diffusion [43]. More recent approaches are based on sparsity regularisation like those of Wavelet shrinkage [2], [8], [29], sparse code shrinkage [10], [15], and techniques relying on compressed sensing [9] and dictionary learning [11]. There are also information theoretical approaches based on the minimum description length principle (MDL) [34].

Model selection is the problem of picking up the best model from a set of candidate models of different complexity. A good model selection technique will find the balance between the goodness of fit and model complexity by implementing the Occam’s razor principle that states that the simplest model is the most likely one of otherwise equal candidates. There are both frequentist and Bayesian methods for model selection such as F tests for nested models [28], Mallow’s C_p [30], Akaike information criterion (AIC) [1], exhaustive search and stepwise/forward/backwards selection procedures [20], crossvalidation [23], Bayes factors of various types [22], Bayesian information criterion BIC [37], the generalisation of AIC and BIC for hierarchical models, the deviance information criterion (DIC) [3], Bayes model averaging [14], and the focused information criterion FIC [13]. An information theoretic approach is the minimum description length (MDL) principle, of which the most recent version is based on the normalised maximum likelihood (NML) criterion [35].

Our approach for separating the protein signal from the micrographs is to estimate the subspace of the vesicle structures in two dimensions and project the micrographs onto the orthogonal complement of this subspace. We chose to model the vesicles structures in two dimensions since this makes the removal simpler from both the statistical modelling and computational point of view, when compared to reconstructing an explicit three dimensional generative model of the vesicle structures including deformations. More precisely, the 2d model, in addition to the vesicle structure, contains the effect of the contrast transfer function (CTF) on the vesicle appearance in the micrographs. The separation of the noise component from the protein signal will be later addressed as part of the reconstruction problem, by the reconstruction algorithm that is beyond the scope of this work.

As the vesicles show a high degree of rotational symmetry, we chose to model the vesicle subspace by the higher order singular value decomposition on the polar coordinate plane, see Fig. 3. Moreover, in contrast to PCA-based approaches such as [6], [19], and [44], HOSVD provides a compact, orthogonal two-dimensional basis characterising the vesicle structure variations since the radial and angular parts are analysed separately. Due to the separation of the radial and angular parts, and the fact that they demonstrate different kinds of variation, we obtain a huge reduction in dimensionality of the model and thus in overfitting, when compared to a standard PCA. To find the best summary of the two-dimensional HOSVD basis patches, we propose another layer onto the HOSVD model by summarising the coefficients by principal component analysis. To select the complexity of the vesicle model, i.e., the number of radial, angular, and summarised components, we use Morozov’s discrepancy principle.

Fig. 3.

The relation between the Cartesian coordinate representation f (x, y) (right) and polar coordinate representation f (ρ, ϕ) (center) in the generation of vesicle HOSVD basis for the normalised, windowed vesicles, as $\rho =\sqrt{x^2+y^2}$ and ϕ = atan2(y, x). Two radial profiles c_i (x, y) = f (ρ, ϕ = ϕ_i), i = 1, 2 evaluated at ${\phi }_1=\frac{50}{251}2\pi ,{\phi }_2=\frac{140}{251}2\pi$ are illustrated as 2d images (left) together with two angular profiles f (ρ = ρ_j , ϕ), j = 1,2 evaluated by ρ₁ = 5 and ρ₂ = 15 (right). The HOSVD basis, constructed in this paper, can be seen as the basis for these kinds of radial and angular structures.

In order to make an accurate modelling of the vesicles, we must properly normalise the vesicle data in position, size and intensity range. The geometric normalisation is carried out by centering the vesicles around the centre point estimates and scaling the sizes using the radius estimates [41].¹ By intensity normalisation we adjust the vesicle signal amplitude and background level to compensate for global illumination and contrast changes, caused by different microscope settings during micrograph acquisition, as illustrated in Fig. 4.

Fig. 4.

A set of size normalised vesicles from different micrographs, illustrating differences in vesicle contrast, background level, and intensity distribution.

The summary of our vesicle removal method is displayed in Fig. 5. The detailed organisation of this paper is as follows. The vesicle modelling theory is described in Section II. The HOSVD model is derived in Section III. Then, the implementation of the model training and vesicle removal is described in Section IV, whereas the model selection method is reported in Section V. The experiments are in Section VI and conclusions in Section VII.

Fig. 5.

Step-by-step illustration of the vesicle structure removal from a micrograph: (a) Vesicle in micrograph, (b) normalised vesicle, (c) windowed, normalised vesicle, (d) projection onto the windowed vesicle subspace, (e) vesicle subtraction as the projection onto the orthogonal complement of the windowed vesicle subspace.

II. Vesicle Modelling

A. Definition of the Objective

In mathematical terms, our goal is to remove the vesicle structures from micrographs, or, to project the micrographs onto the orthogonal complement of the affine subspace describing vesicle structures. In other words, defining the original micrograph as f_micr ∈ $\mathcal{𝒞}$ ⁰(ℝ²), where $\mathcal{𝒞}$ ⁰(ℝ²) is the class of compact, continuous functionals in ℝ², the projection $\mathcal{𝒫}$_⊥ : $\mathcal{𝒞}$ ⁰(ℝ²) → $\mathcal{𝒞}$ ⁰(ℝ²) onto the orthogonal complement is

$$f_{\text{proj}}={\mathcal{𝒫}}_{\perp }{f}_{\text{micr}}.$$ (1)

In practice, our objective is to construct the projection as a sequence of projections over the N vesicles, or,

$${\mathcal{𝒫}}_{\perp }={\mathcal{𝒫}}_{\perp }^{\left(N\right)}\circ {\mathcal{𝒫}}_{\perp }^{(N-1)}\circ \cdots \circ {\mathcal{𝒫}}_{\perp }^{\left(1\right)},$$ (2)

where ${\mathcal{𝒫}}_{\perp }^{\left(n\right)}$ is the projection onto the orthogonal complement of the vesicle n.² We need a separate projection operator for each vesicle as they vary in position, size, shape and intensity. However, the projection operators will coincide in the normalised coordinate frame.

Let ${\mathcal{𝒯}}^{\left(n\right)}\equiv T_{\text{geom}}^{\left(n\right)}\circ T_{\text{win}}^{\left(n\right)}\circ {\mathcal{𝒯}}_{\text{int}}^{\left(n\right)}$ be an affine normalisation operator composed of linear operators $T_{\text{geom}}^{\left(n\right)}$ for the normalisation of the vesicle position and size and $T_{\text{win}}^{\left(n\right)}$ for windowing the vesicle pixels, and the affine intensity normalisation operator ${\mathcal{𝒯}}_{\text{int}}^{\left(n\right)}$. Assume further that $T_{\text{geom}}^{\left(n\right)}$ and ${\mathcal{𝒯}}_{\text{int}}^{\left(n\right)}$ are invertible. Let us define the normalised vesicle image as $f^{\left(n\right)}={\mathcal{𝒯}}^{\left(n\right)}{f}_{\text{micr}}$. The following theorem characterises the nature of the projection operator ${\mathcal{𝒫}}_{\perp }^{\left(n\right)}$.

Theorem 1: Let P_⊥ be the projection onto the linear subspace orthogonal to the affine subspace of windowed and normalised vesicle structures. The projection of the micrograph f_micr onto the orthogonal complement of the affine subspace representing the vesicle structures is

$$\begin{array}{cc}\hfill {\mathcal{𝒫}}_{\perp }^{\left(n\right)}{f}_{\text{micr}}& \equiv {\left({\mathcal{𝒯}}_{\text{int}}^{\left(n\right)}\right)}^{-1}\{{\left(T_{\text{geom}}^{\left(n\right)}\right)}^{-1}{P}_{\perp }\left(f^{\left(n\right)}-g_0\right)\phantom{\}}\hfill \\ \hfill & \phantom{\{}+\left(\left(I-T_{\text{win}}^{\left(n\right)}\right)\circ {\mathcal{𝒯}}_{\text{int}}^{\left(n\right)}\right)f_{\text{micr}}\},\hfill \end{array}$$ (3)

where g₀ is the offset of the affine subspace, and $I-T_{\text{win}}^{\left(n\right)}$ is the windowing operator corresponding to the backgroud window $1-w^{\left(n\right)}$.

Proof: Let $T_{\text{win}}^{\left(n\right)}f\equiv w^{\left(n\right)}f,f\in {\mathcal{𝒞}}^0\left({\mathbb{ℝ}}^2\right)$, where $0\le w^{\left(n\right)}(x,y)\le 1,\forall x,y\in \mathbb{ℝ}$. The window functional w⁽ⁿ⁾ divides the micrograph into the foreground and background of the vesicle n such that

$$\begin{array}{cc}\hfill f_{\text{micr}}\equiv {\left({\mathcal{𝒯}}_{\text{int}}^{\left(n\right)}\right)}^{-1}& \{w^{\left(n\right)}\left({\mathcal{𝒯}}_{\text{int}}^{\left(n\right)}{f}_{\text{micr}}\right)\phantom{\}}\hfill \\ \hfill & \phantom{\{}+\left(1-w^{\left(n\right)}\right)\left({\mathcal{𝒯}}_{\text{int}}^{\left(n\right)}{f}_{\text{micr}}\right)\}\hfill \end{array}$$ (4)

where 1 – w⁽ⁿ⁾ is the background window. This is equivalent

$$\begin{array}{cc}\hfill & {\left({\mathcal{𝒯}}_{\text{int}}^{\left(n\right)}\right)}^{-1}\left\{\left(T_{\text{win}}^{\left(n\right)}\circ {\mathcal{𝒯}}_{\text{int}}^{\left(n\right)}\right)f_{\text{micr}}+\left(\left(I-T_{\text{win}}^{\left(n\right)}\right)\circ {\mathcal{𝒯}}_{\text{int}}^{\left(n\right)}\right)f_{\text{micr}}\right\}\hfill \\ \hfill & ={\left({\mathcal{𝒯}}_{\text{int}}^{\left(n\right)}\right)}^{-1}\{{\left(T_{\text{geom}}^{\left(n\right)}\right)}^{-1}\left(T_{\text{geom}}^{\left(n\right)}\circ T_{\text{win}}^{\left(n\right)}\circ {\mathcal{𝒯}}_{\text{int}}^{\left(n\right)}\right)f_{\text{micr}}\phantom{\}}\hfill \\ \hfill & +\phantom{\{}\left(\left(I-T_{\text{win}}^{\left(n\right)}\right)\circ {\mathcal{𝒯}}_{\text{int}}^{\left(n\right)}\right)f_{\text{micr}}\}\hfill \\ \hfill & ={\left({\mathcal{𝒯}}_{\text{int}}^{\left(n\right)}\right)}^{-1}\left\{{\left(T_{\text{geom}}^{\left(n\right)}\right)}^{-1}{f}^{\left(n\right)}+\left(\left(I-T_{\text{win}}^{\left(n\right)}\right)\circ {\mathcal{𝒯}}_{\text{int}}^{\left(n\right)}\right)f_{micr}\right\}.\hfill \end{array}$$ (5)

Now, let us make a subspace approximation for the normalised, windowed image $f^{\left(n\right)}$, such that

$$f^{\left(n\right)}={\widehat{f}}^{\left(n\right)}+r^{\left(n\right)},$$ (6)

where ${\widehat{f}}^{\left(n\right)}=g_0+P\left(f^{\left(n\right)}-g_0\right)$, and P is the projection onto the linear subspace of the mean corrected vesicle structures. The residual after the removal of ${\widehat{f}}^{\left(n\right)}$ is equivalent to

$$\begin{array}{cc}\hfill r^{\left(n\right)}& =f^{\left(n\right)}-{\widehat{f}}^{\left(n\right)}=\left(f^{\left(n\right)}-g_0\right)-P\left(f^{\left(n\right)}-g_0\right)\hfill \\ \hfill & =P_{\perp }\left(f^{\left(n\right)}-g_0\right)\hfill \end{array}$$ (7)

where $P_{\perp }=I-P$. The removal of the vesicle is thus obtained by replacing $f^{\left(n\right)}\text{by}{r}^{\left(n\right)}=f^{\left(n\right)}-{\widehat{f}}^{\left(n\right)}$ in (5) that yields the projection operator

$$\begin{array}{cc}\hfill {\mathcal{𝒫}}_{\perp }^{\left(n\right)}{f}_{\text{micr}}\equiv {\left({\mathcal{𝒯}}_{\text{int}}^{\left(n\right)}\right)}^{-1}& \{{\left(T_{\text{geom}}^{\left(n\right)}\right)}^{-1}{P}_{\perp }\left(f^{\left(n\right)}-g_0\right)\phantom{\}}\hfill \\ \hfill & +\phantom{\{}\left(\left(I-T_{\text{win}}^{\left(n\right)}\right)\circ {\mathcal{𝒯}}_{\text{int}}^{\left(n\right)}\right)f_{\text{micr}}\}\hfill \end{array}$$ (8)

and the claim follows.

B. Vesicle Normalisation

In this section, we first describe the normalisation of the vesicle position and size and the windowing. Thereafter, we describe the normalisation of the intensity range to remove illumination differences.

1) Shape Normalisation: Let $f_{\text{norm}}^{\left(n\right)}$ be the intensity normalised micrograph ${\mathcal{𝒯}}_{\text{int}}^{\left(n\right)}{f}_{\text{micr}}$. The composite operator $T_{\text{geom}}^{\left(n\right)}\circ T_{\text{norm}}^{\left(n\right)}:f_{\text{norm}}^{\left(n\right)}\mapsto f^{\left(n\right)}$ representing the geometric normalisation and windowing as Fig. 6 illustrates, is defined as

$$\begin{array}{cc}\hfill f^{\left(n\right)}(x,y)& =f_{\text{norm}}^{\left(n\right)}(x_{\text{micr}},y_{\text{micr}})\hfill \\ \hfill & \times w^{\left(n\right)}(x_{\text{micr}}-x_{\text{micr}}^{\left(n\right)},y_{\text{micr}}-y_{\text{micr}}^{\left(n\right)}),\hfill \end{array}$$ (9)

where $(x_{\text{micr}}^{\left(n\right)},y_{\text{micr}}^{\left(n\right)})$ is the vesicle center in the micrograph, w⁽ⁿ⁾ the window functional, and

$$\left(\begin{array}{c}\hfill x_{\text{micr}}\hfill \\ \hfill y_{\text{micr}}\hfill \end{array}\right)=\frac{R^{(n)}}{\underset{n^{\prime }}{\text{max}}{R}^{\left(n^{\prime }\right)}}\left(\begin{array}{c}\hfill x\hfill \\ \hfill y\hfill \end{array}\right)+\left(\begin{array}{c}\hfill x_{\text{micr}}^{\left(n\right)}\hfill \\ \hfill y_{\text{micr}}^{\left(n\right)}\hfill \end{array}\right).$$ (10)

The ratio $R^{\left(n\right)}∕\underset{{\text{n}}^{\prime }}{\text{max}}{R}^{\left(n^{\prime }\right)}$ is the relative vesicle radius in the original image, normalised by the maximum radius over the n′ training vesicles.

Fig. 6.

(a) A normalised vesicle; (b) the corresponding Tukey window; (c) the windowed vesicle. The vesicle signal is captured by the window while a small overlap of other vesicles may be visible on the transition region.

Representing the normalised vesicle in polar coordinates $f^{\left(n\right)}(\rho ,\phi )\equiv f^{\left(n\right)}(\rho (x,y),\phi (x,y))$, we define the vesicle window as

$$\begin{array}{cc}\hfill & w^{\left(n\right)}(x_{\text{micr}}-x_{\text{micr}}^{\left(n\right)},y_{\text{micr}}-y_{\text{micr}}^{\left(n\right)})\hfill \\ \hfill & \equiv w_{\text{Tuk}}^{\left(n\right)}\left(r(x_{\text{micr}}-x_{\text{micr}}^{\left(n\right)},y_{\text{micr}}-y_{\text{micr}}^{\left(n\right)})\right)\hfill \end{array}$$ (11)

where

$$\begin{array}{cc}\hfill & w_{\text{Tuk}}^{\left(n\right)}\left(r\right)\hfill \\ \hfill & =\{\begin{array}{cc}1,\hfill & \text{if}r\le R^{\left(n\right)}\hfill \\ \frac{1}{2}+\frac{\text{cos}\pi \alpha (r-R^{\left(n\right)})}{2},\hfill & \text{if}{R}^{\left(n\right)}<r<R^{\left(n\right)}+\frac{1}{\alpha }\hfill \\ 0,\hfill & \text{otherwise};\hfill \end{array}\phantom{\}}\hfill \end{array}$$ (12)

is the Tukey window. The parameter α controls the steepness of transition region of the window (see Fig. 6).

2) Intensity Normalisation: In the intensity normalisation, we aim at recovering the differences in contrast (Fig. 4) and background level (Fig. 7). We define the pixel intensity normalisation as the operator ${\mathcal{𝒯}}_{\text{int}}^{\left(n\right)}:f_{\text{micr}}\mapsto f_{\text{norm}}^{\left(n\right)}$ as

$$f_{\text{norm}}^{\left(n\right)}(x,y)=\frac{f_{\text{micr}}(x,y)-b(x,y)}{s(x,y)}$$ (13)

where b(x , y) is the background reference level and s(x, y) the vesicle signal level.

Fig. 7.

An example how the background level drifts on a micrograph.

Due to global changes of the background level (Fig. 7), we use a local median estimate for each vesicle, serving as a robust reference level estimator for the vesicle signal. For the vesicle n, the background level is thus estimated as

$${\widehat{b}}^{\left(n\right)}=\underset{(x^{\prime },y^{\prime })\in {\mho }^{\left(n\right)}}{\text{median}}{f}_{\text{micr}}(x^{\prime }-x_{\text{micr}}^{\left(n\right)},y^{\prime }-y_{\text{micr}}^{\left(n\right)})$$ (14)

where the local background area is defined as

$$\begin{array}{cc}\hfill {\mho }^{\left(n\right)}=& \{(x^{\prime },y^{\prime })\mid w_{\text{loc}}(x^{\prime }-x_{\text{micr}}^{\left(n\right)},y^{\prime }-y_{\text{micr}}^{\left(n\right)})=1\phantom{\}}\hfill \\ \hfill & \wedge \phantom{\{}{w}_{\text{bg}}(x,y)=1\},\hfill \end{array}$$ (15)

whereas the local support

$$\begin{array}{cc}\hfill & w_{\text{loc}}(x^{\prime }-x^{\left(n\right)},y^{\prime }-y^{\left(n\right)})\hfill \\ \hfill & =\{\begin{array}{cc}1,\hfill & \text{if}d(x^{\prime }-x_{\text{micr}}^{\left(n\right)},y^{\prime }-y_{\text{micr}}^{\left(n\right)})<R_{\text{loc}}\hfill \\ 0,\hfill & \text{otherwise},\hfill \end{array}\phantom{\}}\hfill \end{array}$$ (16)

where $R_{\text{loc}}>R^{\left(n\right)}+\frac{1}{\alpha }$ is the radius of the local support; and the background is windowed by

$$w_{\text{bg}}(x,y)=\{\begin{array}{cc}1\hfill & \text{if}\forall nd(x-x_{\text{micr}}^{\left(n\right)},y-y_{\text{micr}}^{\left(n\right)})\ge R^{\left(n\right)}+\frac{1}{\alpha }\hfill \\ 0\hfill & \text{otherwise}\hfill \end{array}\phantom{\}}$$ (17)

The local background is illustrated in Fig. 8, where the radius of the local support is set to half the width of the image.

Fig. 8.

(a) A vesicle image; (b) the corresponding local background window; (c) windowed background pixels from which the background level is estimated.

Finally, the vesicle signal level is robustly estimated by the median absolute deviation (MAD) estimator or

$$\begin{array}{cc}\hfill {\widehat{s}}^{\left(n\right)}& =\underset{(x^{\prime },y^{\prime })\in {\Omega }^{\left(n\right)}}{\text{median}}\mid f_{\text{micr}}(x^{\prime }-x_{\text{micr}}^{\left(n\right)},y^{\prime }-y_{\text{micr}}^{\left(n\right)})\phantom{\mid }\hfill \\ \hfill & \times \phantom{\mid }{w}^{\left(n\right)}(x^{\prime }-x_{\text{micr}}^{\left(n\right)},y^{\prime }-y_{\text{micr}}^{\left(n\right)})-{\widehat{b}}^{\left(n\right)}\mid \hfill \end{array}$$ (18)

where Ω⁽ⁿ⁾ is the support of the vesicle n.

C. Polar Coordinate Representation

In the construction of the projection operator (1), (3), our next goal is to construct the vesicle model ${\widehat{f}}^{\left(n\right)}$ for vesicles n = 1, 2, …, N by statistical modelling. We are searching for a reasonable fit in the sense of the energy functional

$$E=\sum _{n=1}^N{\int }_{{\Omega }^{\left(n\right)}}{\left(f^{\left(n\right)}\left(\mathbf{r}\right)-{\widehat{f}}^{\left(n\right)}\left(\mathbf{r}\right)\right)}^{2}d\mathbf{r},$$ (19)

where r = (x , y) is a position in the normalised image and the support Ω⁽ⁿ⁾ of each vesicle f ⁽ⁿ⁾ is known. We seek for the models that provide both a good fit, but do not overfit, and are compact, i.e., they provide a reasonable energy level and small number of parameters and hence separate vesicle signal from the background noise.

We assume that the vesicles lie in an orthogonal subspace to the noise subspace on the image plane. So our aim is to find an orthonormal basis ${\left\{g_k\left(\mathbf{r}\right)\right\}}_{k=1}^K$ spanning the vesicle subspace, or,

$$\langle g_k,g_k\rangle =1,k=1,\dots ,K$$ (20)

and

$$\langle g_k,g_l\rangle =0\text{for}\text{all}k\ne l.$$ (21)

Vesicles have a high level of symmetry in the polar coordinates that should be taken into account to make the representation compact. The least squares analysis in Cartesian (x , y) and polar coordinates (ρ, ϕ) are equivalent in the following sense.

Lemma 2: The L₂ minimisation on the image domain is equivalent to L₂ minimisation on a weighted polar coordinate frame, where the weighting function is $\sqrt{\rho (x,y)}$.

Proof: An L₂ approximation $\widehat{f}(x,y)$ of the image f (x , y) = f (x (ρ, ϕ), y(ρ, ϕ)) = f (ρ, ϕ) minimises

$$\begin{array}{cc}\hfill E& =\int \int (f-\widehat{f})dxdy\hfill \\ \hfill & =\int \int \rho {(f(\rho ,\phi )-\widehat{f}(\rho ,\phi ))}^{2}d\rho d\phi \hfill \\ \hfill & =\int \int {(\sqrt{\rho }f(\rho ,\phi )-\sqrt{\rho }\widehat{f}(\rho ,\phi ))}^{2}d\rho d\phi \hfill \end{array}$$ (22)

which is the L₂-minimisation between $\sqrt{\rho }f(\rho ,\phi )$ and $\sqrt{\rho }\widehat{f}(\rho ,\phi )$.

The lemma gives us the freedom to make the analysis in the polar coordinates as long as we take the weighting $\sqrt{\rho }$ into consideration.

To make the representation compact, we assume that the basis is separable in the polar coordinates or

$$g_k\left(\mathbf{r}(\rho ,\phi )\right)={\tilde{u}}_k^{\left(1\right)}\left(\rho \right)u_k^{\left(2\right)}\left(\phi \right)\equiv \frac{u_k^{\left(1\right)}\left(\rho \right)}{\sqrt{\rho }}{u}_k^{\left(2\right)}\left(\phi \right).$$ (23)

The separability implies the following useful lemma, since we want to generate an orthogonal basis in the vesicle subspace.

Lemma 3: Let ${\left\{g_k\right\}}_{k=1}^K$ be a separable basis in the polar coordinates. The basis is orthogonal if and only if either the radial or the angular part of the basis is orthogonal.

Proof: For k ≠ l, from the separability it follows

$$\begin{array}{cc}\hfill \langle g_k,g_l\rangle & =\left(\int \rho {\tilde{u}}_k^{\left(1\right)}\left(\rho \right){\tilde{u}}_l^{\left(1\right)}\left(\rho \right)d\rho \right)\hfill \\ \hfill & \times \left(\int u_k^{\left(2\right)}\left(\phi \right)u_l^{\left(2\right)}\left(\phi \right)d\phi \right)\hfill \\ \hfill & =\langle \sqrt{\rho }{\tilde{u}}_k^{\left(1\right)},\sqrt{\rho }{\tilde{u}}_l^{\left(1\right)}\rangle \langle u_k^{\left(2\right)},u_l^{\left(2\right)}\rangle \hfill \\ \hfill & =\langle u_k^{\left(1\right)},u_l^{\left(1\right)}\rangle \langle u_k^{\left(2\right)},u_l^{\left(2\right)}\rangle .\hfill \end{array}$$ (24)

where $u_k^{\left(1\right)}\left(\rho \right)$ is a weighted radial basis function and $u_k^{\left(2\right)}\left(\phi \right)$ an angular basis function. The right hand side vanishes if and only if either the radial or angular part is orthogonal.

The message here is thus the follows.

Corollary 4: Aiming at the least squares fit and an orthogonal basis for the vesicle subspace on the image plane, it is equivalent to inspect the √ρ weighted vesicle images in the polar coordinate plane and construct an orthogonal bases for the radial and angular part separately.³

Assuming $f^{\left(n\right)}(x,y)$ is a normalised, windowed vesicle image (9) we define the operator $T_{\rho \phi }^{\left(n\right)}:f^{\left(n\right)}\mapsto f_{\rho \phi }^{\left(n\right)}$ transform f ⁽ⁿ⁾(x , y) into the weighted, polar coordinates as

$$f_{\rho \phi }^{\left(n\right)}(\rho ,\phi )=\sqrt{\rho }{f}^{\left(n\right)}(\rho ,\phi )=\sqrt{\rho (x,y)}{f}^{\left(n\right)}(x,y).$$ (25)

In the following section, we describe how we construct orthogonal bases using the higher-order singular value decomposition (HOSVD).

III. HOSVD

A. Basic Model

Let us have the discretised vesicles stored in the three-way array 𝒜 ∈ ℝ^{I₁ ×I₂ ×I₃} in the weighted polar coordinate frame; let us also introduce the reference array 𝒜₀ ∈ ℝ^{I₁ ×I₂ ×I₃}, where the dimensions d = 1, 2, 3 correspond to the weighted radial, angular, and vesicle index, respectively, and I₃ ≡ N represents the number of vesicle training images. The structure of 𝒜 is illustrated in Fig. 9. We look for the approximation $\widehat{𝒜}$ for 𝒜 − 𝒜₀. We are going to use the following theorem.

Fig. 9.

The three-way array 𝒜 ∈ ℝ^{I₁×I₂ ×I₃} illustrated with three vesicle images and their unfoldings 𝒜₍₁₎ and 𝒜₍₂₎. Each column in 𝒜₍₁₎ is a radial profile evaluated at a fixed angle in a vesicle image. Each column in 𝒜₍₂₎ is an angular profile evaluated at a fixed radius in a vesicle image.

Theorem 5: There is a decomposition

$$𝒜-{𝒜}_0=𝒮{\times }_1{\mathbf{U}}^{\left(1\right)}{\times }_2{\mathbf{U}}^{\left(2\right)}{\times }_3{\mathbf{U}}^{\left(3\right)},$$ (26)

where U^(d) are orthonormal I_d × I_d matrices, d = 1, 2, 3, ×_d is the d-way product, and 𝒮 ∈ ℝ^{I₁ ×I₂ ×I₃} is the core array whose subarrays are (1) all-orthogonal, i.e., for all possible values of $d,\alpha ,\beta ,\alpha \ne \beta ,\langle {𝒮}_{i_d=\alpha },{𝒮}_{i_d=\beta }\rangle =0$; (2) sorted so that the d^th order singular values ${\sigma }_{\alpha }^{\left(d\right)}=\parallel {𝒮}_{i_d=\alpha }{\parallel }_{\text{fro}}$ are sorted as ${\sigma }_1^{\left(d\right)}\ge {\sigma }_2^{\left(d\right)}\ge \cdots \ge {\sigma }_{I_d}^{\left(d\right)},d=1,2,3$.

The proof is in [26]. This decomposition is known as higher-order singular value decomposition (HOSVD). It is useful due to the following approximation property, which is the higherorder counter part to the approximation property of the SVD.

Theorem 6: Let $\widehat{𝒜}$ be the array obtained by truncating the basis formed by the column vectors of ${\sigma }_{{\tilde{I}}_d+1}^{\left(d\right)},{\sigma }_{{\tilde{I}}_d+2}^{\left(d\right)},\dots ,{\sigma }_{I_d}^{\left(d\right)},d=1,2,3$ Then,

$$\parallel (𝒜-{𝒜}_0)-\widehat{𝒜}{\parallel }_{\text{fro}}\le \sum _{d=1}^3\sum _{i_d={\tilde{I}}_d+1}^{I_d}{\sigma }_{i_d}^{\left(d\right)}.$$ (27)

The proof can likewise found in [26]. Though the HOSVD approximation is good enough for practical purposes, there may be another basis that provides tighter approximation in the least square sense with the same number of basis vectors ${\tilde{I}}_d,d=1,2,3$, retained.

The practical computation of the HOSVD is easy with the help of the ordinary SVD. The HOSVD can be implemented by making a sequence of matrix unfoldings for 𝒜 − 𝒜₀ and computing the singular value decomposition for each. In detail, let the d^th unfolding be denoted as A_(d) − A_0,(d), d = 1, 2, 3. The d = 1 and d = 2 unfoldings are illustrated in Fig. 9. A₍₁₎ allows for the analysis of the weighted radial components and A₍₂₎ for the angular components, one component a column, independently of any other variation in the remaining dimensions of the data. The d^th mode singular matrices U^(d) are found as the left singular matrices of the corresponding d^th mode matrix unfolding, or A_(d) − A_0,(d) = U^(d)Σ^(d)V^(d)T. We can alternatively state as follows.

Corollary 7: The d-fold singular vectors are the eigenvectors of the d-fold correlation matrix

$${\mathbf{R}}^{\left(d\right)}=\frac{1}{I_d}({\mathbf{A}}_{\left(d\right)}-{\mathbf{A}}_{0,\left(d\right)}){({\mathbf{A}}_{\left(d\right)}-{\mathbf{A}}_{0,\left(d\right)})}^{\text{T}}$$ (28)

Corollary 7 reveals the relation of the HOSVD to the principal component analysis (PCA), and the reason why we introduced the reference array 𝒜₀. In principal component analysis one computes the eigenvectors of the sample covariance matrix where the observations have been centred around the mean observation. In the multidimensional case, it is not possible since there is generally no such a reference array that would produce the means of each unfoldings by just taking the corresponding unfolding of the reference array. However, we may select the reference array so that its d = 1 and d = 2 unfoldings are consistent, by setting the reference array to be a stack of the mean vesicle copies computed over the vesicle population d = 3 that is a good compromise.

The vesicle bases are thus generated using the d = 1 and d = 2 unfoldings of the array 𝒜 − 𝒜₀ to compute the weighted radial and angular basis vectors as the columns of U⁽¹⁾ and U⁽²⁾. We then truncate the corresponding d-mode singular values by retaining ${\tilde{I}}_1\text{and}{\tilde{I}}_2$ singular vectors in ${\tilde{\mathbf{U}}}^{\left(1\right)}\text{and}{\tilde{\mathbf{U}}}^{\left(2\right)}$, respectively. We generate the orthogonal basis G = (g_k), corresponding to (20), where k = k(i₁, i₂) and $i_1=1,2,\dots ,{\tilde{I}}_1,i_2=1,2,\dots {\tilde{I}}_2$, are the column indices of ${\tilde{\mathbf{U}}}^{\left(1\right)}\text{and}{\tilde{\mathbf{U}}}^{\left(2\right)}$, respectively. In other words, all weighted radial basis vectors are paired with all angular basis vectors (see Fig. 10). A vesicle is hence modelled as

$$\widehat{\mathbf{f}}={\mathbf{g}}_0+\mathbf{G}\xi ,$$ (29)

where g₀ is the mean of all normalised, discretised images ${\mathbf{f}}^{\left(n\right)},\xi ={\mathbf{G}}^{\text{T}}({\mathbf{f}}^{\left(n\right)}-{\mathbf{g}}_0)$ are the coordinates at the vesicle in the basis formed by the column vectors of G.

Fig. 10.

Learnt vesicle basis visualised in the Cartesian coordinate frame sorted in the order of importance: (top row) pure radial components; (left column) pure angular components; (remaining basis elements) respective combinations of the pure radial and angular components.

B. Hierarchical Model

The basic HOSVD model does not directly reveal whether any of the generated 2d bases images is irrelevant and could be dropped. We solve this problem by adding a PCA layer to the HOSVD model as follows.

Let ${\xi }^{\left(i_3\right)}$ be the coefficient vector of the normalised vesicle image f ⁽ⁱ³⁾. For all coefficient vectors, we compute the mean $\stackrel{‒}{\xi }$ and the covariance matrix $\mathbf{C}={\sum }_{i_3}({\xi }^{\left(i_3\right)}-\stackrel{‒}{\xi }){({\xi }^{\left(i_3\right)}-\stackrel{‒}{\xi })}^{\text{T}}$. We may now write

$$\xi =\stackrel{‒}{\xi }+\sum _{k=1}^K{\lambda }_k<{\mathbf{v}}_k,\xi >{\mathbf{v}}_k=\stackrel{‒}{\xi }+\mathbf{V}\Lambda {\mathbf{V}}^{\text{T}}\xi$$ (30)

where λ_k, v_k , k = 1, 2, … , K are the eigenvalues and the eigenvectors of C, all ordered in the descending order of the eigenvalues. Substituting (30) into (29) yields the ordered model,

$$\begin{array}{cc}\hfill \widehat{\mathbf{f}}& ={\text{g}}_0+\mathbf{G}(\stackrel{‒}{\xi }+\mathbf{V}\Lambda {\mathbf{V}}^{\text{T}}\xi )\hfill \\ \hfill & ={\text{g}}_0+\mathbf{G}\xi +\mathbf{G}\mathbf{V}\Lambda {\mathbf{V}}^{\text{T}}\xi \hfill \\ \hfill & ={\text{g}}_0^{\prime }+{\mathbf{G}}^{\prime }{\xi }^{\prime }\hfill \end{array}$$ (31)

for the combined model offset

$${\text{g}}_0^{\prime }={\text{g}}_0+\mathbf{G}\stackrel{‒}{\xi }$$ (32)

where $\mathbf{G}\stackrel{‒}{\xi }$ is the vesicle model estimate of the mean coefficient, ${\xi }^{\prime }=\Lambda {\mathbf{V}}^{\text{T}}\xi$ and G^′ = GV. In other words, ${\mathbf{g}}_0^{\prime }$ represents the offset and G^′ a new set of ordered basis vectors. The $\tilde{K}$ smallest eigenvalues and the associated new basis vectors ${\tilde{\mathbf{G}}}^{\prime }$ can then be truncated to achieve a compressed model. The construction of the PCA layer is illustrated in Fig. 11.

Fig. 11.

Construction of the second layer for the HOSVD model. (Left) The most descriptive paired 2d bases are selected by pairing the most descriptive radial and angular bases. (Right) The new basis is generated by PCA as the linear combination of the selected paired components while the least significant new basis elements are truncated.

IV. Implementation

In Section IV-A, we describe the complete model training procedure, after which, in Section IV-B, we report how vesicles are removed from micrographs.

A. Training

Given a set of micrographs containing particle-free vesicles, with known vesicle center positions and radii, the model is trained as follows. The vesicle images are first intensity normalised (13), windowed, shape normalised (9), and transformed onto the polar coordinate plane (25). Then, the polar coordinate patches are stacked into the three-way array 𝒜. The array 𝒜 is decomposed by the higher-order singular value decomposition (26) of which the smallest singular vectors corresponding to the radial and angular dimensions are truncated. As the last step, the hierarchical model (31) is constructed and the smallest singular vectors truncated to form the final basis ${\tilde{\mathbf{G}}}^{\prime }$. The whole training algorithm is summarised in Algorithm 1.

B. Vesicle Removal

Given the trained affine vesicle basis $\left({\mathbf{g}}_0^{\prime },{\tilde{\mathbf{G}}}^{\prime }\right)$ and a novel micrograph with vesicle center positions and radii, the vesicle structures are removed as follows. Each vesicle is intensity normalised, windowed and shape normalised, after which they are projected onto the residual subspace by the orthogonal projection matrix (cf. (7)),

$${\mathbf{P}}_{\perp }=\mathbf{I}-\mathbf{P}=\mathbf{I}-{\tilde{\mathbf{G}}}^{\prime }{\tilde{\mathbf{G}}}^{\prime T}$$ (33)

that yields the residual

$${\mathbf{r}}^{\left(n\right)}={\mathbf{P}}_{\perp }\left({\mathbf{f}}^{\left(n\right)}-{\mathbf{g}}_0^{\prime }\right).$$ (34)

The vesicle removal is achieved by reversing the shape normalisation for the residual and superimposing the result with the background windowed signal, or,

$${\tilde{\mathbf{f}}}^{\left(n\right)}={{\mathbf{T}}_{\text{geom}}^{\left(n\right)}}^{-1}\left({\mathbf{r}}^{\left(n\right)}\right)+{\mathbf{b}}^{\left(n\right)}$$ (35)

after which the reverse intensity normalisation follows. The complete vesicle removal procedure is described in Algorithm 2, and illustrated in Fig. 12.

Fig. 12.

Illustration of the vesicle removal in the function of the number of basis components. (a) The normalised vesicle; (b) the mean subtracted. Projection onto the affine residual subspace with (c) 2, (d) 10, (e) 37, and (f) 42 basis elements.

V. Model Selection

The remaining task is to determine the complexity of the vesicle model, that is, to find the appropriate number of components $({\tilde{I}}_1,{\tilde{I}}_2,\tilde{K})$. We assume i.i.d. additive noise, i.e., the observed normalised vesicle follows the model

$${\mathbf{f}}^{\left(n\right)}={\widehat{\mathbf{f}}}^{\left(n\right)}+ϵ$$ (36)

where ${\widehat{\mathbf{f}}}^{\left(n\right)}$ is the noise-free vesicle and r the noise vector. Thus, in the model selection we aim at a model that has enough parameters to describe the structure variations in the vesicle population so that it does not underfit the data. However, the model should not be too complex so that it is able to represent the unrelated background noise, i.e., the model should not overfit the data.

To solve the model selection problem, we use two distinct principles: Morozov’s discrepancy principle together with the Occam razor principle. According to Morozov’s discrepancy principle [21], the right complexity is achieved when the residual has the identical level to the assumed noise level. Typically in ill-posed problems, the application of this principle however yields an ambiguity, since the solution yielding the desired level is generally not unique. To resolve this ambiguity, we apply the Occam razor principle that states that among multiple solutions, the simplest one is the most likely, i.e., we select the simplest solution at the desired residual noise level.

To use Morozov’s discrepancy principle, we estimate the background noise level ${ϵ}_{\text{noise}}^{\left(n\right)}$ by using (14) in the discrete form. The residual level ${ϵ}_{\text{res}}^{\left(n\right)}$ is estimated similarly to (18) with the difference that the signal is replaced by only the residual part. As the criterion, derived from Morozov’s discrepancy principle, we use the mean difference

$$d({\tilde{I}}_1,{\tilde{I}}_2,\tilde{K})={\overline{ϵ}}_{\text{noise}}-{\overline{ϵ}}_{\text{res}}.$$ (37)

between the estimated background and residual noise levels. As illustrated in Fig. 13, the case $d({\tilde{I}}_1,{\tilde{I}}_2,\tilde{K})>0$ represents models which have been overfitted, whereas the case $d({\tilde{I}}_1,{\tilde{I}}_2,\tilde{K})<0$ corresponds to underfitted models. The desired complexity would be achieved by the models on the isosurface $d({\tilde{I}}_1,{\tilde{I}}_2,\tilde{K})$ = 0, i.e., when ${\overline{ϵ}}_{\text{noise}}={\overline{ϵ}}_{\text{res}}$, assuming that the parameters were continuous variables. In practice, since the model parameters are discrete, we associate the isosurface by the set of parameter configurations for which the distance from the continuous, interpolated approximation of the isosurface is smaller than unity in the infinity norm.

Fig. 13.

Sketch of the behaviour of the mean difference $d={\overline{ϵ}}_{\text{noise}}-{\overline{ϵ}}_{\text{res}}$ between the background noise and residual noise with different number of basis elements. The cases d > 0 represent overfitted models, d < 0 underfitted models, and, d = 0 the models with the desired complexity.

To finish the model selection procedure, by using the Occam razor principle, we select the simplest solution from the family of solutions at the desired noise level. To select the simplest model, we need to characterise the effective number of parameters in the model. Using the two layer HOSVD model, the number of effective parameters is characterised by the following theorem.

Theorem 8: The number of parameters of the two-layer HOSVD model is

$$\begin{array}{cc}\hfill c({\tilde{I}}_1,{\tilde{I}}_2,\tilde{K})& =I_1{I}_2+{\tilde{I}}_1{\tilde{I}}_2+\tilde{K}+{\tilde{I}}_1{\tilde{I}}_1-\frac{{\tilde{I}}_1+1}{2}{\tilde{I}}_1\hfill \\ \hfill & +{\tilde{I}}_2{\tilde{I}}_2-\frac{{\tilde{I}}_2+1}{2}{\tilde{I}}_2+{\tilde{I}}_1{\tilde{I}}_2\tilde{K}-\frac{\tilde{K}+1}{2}\tilde{K}\hfill \end{array}$$ (38)

where I₁ and I₂ are the number of discrete radii and the number of discrete angles in a vesicle image, respectively; ${\tilde{I}}_1$ and ${\tilde{I}}_2$ are the corresponding numbers of retained components after truncation; and $\tilde{K}$ is the number of retained second layer basis vectors.

Proof: Starting from the first layer, the vesicle population mean has I₁ I₂ parameters. The 2d radial basis can be parameterised by parameterising the one-dimensional basis {u_i1 }. Since the basis is orthonormal, it has ${\sum }_{i_1=1}^{{\tilde{I}}_1}(I_1-i_1)$ degrees of freedom. Similarly, the 2d angular basis has ${\sum }_{i_2=1}^{{\tilde{I}}_2}(I_2-i_2)$ degrees of freedom. The second layer offset has ${\tilde{I}}_1{\tilde{I}}_2$ parameters, and the second layer basis ${\sum }_{k=1}^{\tilde{K}}({\tilde{I}}_1{\tilde{I}}_2-k)$ parameters. The retained second layer eigenvalues have $\tilde{K}$ parameters. The number of parameters, and thus the complexity $c({\tilde{I}}_1,{\tilde{I}}_2,\tilde{K})$ of the whole model is obtained by summing up the number of parameters above, and the claim follows.

The complete model selection procedure is described in Algorithm 3.

VI. Experiments

In the experiments, we used the transmission electron microscope training data set consisting of ten 2048 × 2048 micrographs with all together N = 1143 vesicles taken by different CTF settings. The model selection was performed with N = 436 vesicles extracted from five additional, independent micrographs, where some vesicles had a protein particle attached. The data sets contained the vesicle center positions and radii, automatically detected as reported in [27]. The bright aura around the vesicle signal is influenced by both the presence of vesicle neighbours and the microscope settings, see Fig. 21. The vesicle window radius was thus estimated so that it fully covers a few pixels of the bright vesicle aura, after which the transition region of the window brings the signal amplitude visually reasonably onto the background level (Fig. 6), to include the aura in the model while limiting influence by vesicle neighbours. This was achieved by the setting $\alpha =\frac{1}{12}$, for the Tukey window (12). The normalised image pixel size was determined by the largest vesicle in the database. The vesicle models were trained as described in Alg. 1. The training and testing were implemented in Matlab. The computations were performed with a conventional laptop computer, where the training took 110s, model selection 1h 20 min, and the vesicle removal 9–12s/micrograph.

Fig. 21.

Vesicle removal result for a full micrograph. (a) Original micrograph; (b) vesicles removed by the modified RPCA model with 78 low rank components; (c) vesicles removed by the proposed method with the automatically selected model $({\tilde{I}}_1,{\tilde{I}}_2,\tilde{K})$ = (6, 5, 27). Vesicle center positions are marked with blue dots and all images have been post-processed by a low-pass filter to enhance structural details.

The model selection was carried out as described in Alg. 3, over the model class ${\tilde{I}}_1\in \{1,2,\dots ,15\},{\tilde{I}}_2\in \{1,2,\dots ,15\},\tilde{K}\in \{1,2,\dots ,{\tilde{I}}_1{\tilde{I}}_2\}$. As the ground truth for model selection, we manually evaluated the most relevant combinations of $({\tilde{I}}_1,{\tilde{I}}_2,\tilde{K})$ and picked up the combination (7, 7, 40) that showed visually the best vesicle removal result. The estimated continuous extension of the mean difference d_ext (i₁, i₂, k) between the background noise and residual noise is illustrated in Fig. 14(a). In other words, the isosurface in Fig. 14(a) intersects all zero-valued residuals, corresponding to valid vesicle models, in the volume of residuals measured for each model complexity $({\tilde{I}}_1,{\tilde{I}}_2,\tilde{K})$ in the experiment.

Fig. 14.

(a) The isosurface where the mean residual noise level equals to the mean background noise level or d_ext (i₁, i₂ , k) = 0; the colours outside the isosurface indicate the residual level. (b) Illustration of the model selection result; the right complexity is achieved when the isosurfaces d_ext (i₁, i₂, k) = 0 (in colour) and c(i₁ , i₂ , k) = c_min (gray) meet at the tangent point $({\tilde{I}}_1,{\tilde{I}}_2,\tilde{K})=(6,5,27)$; the surface colour indicates the complexity.

Our model selection method yielded the result $({\tilde{I}}_1,{\tilde{I}}_2,\tilde{K})$ = (6, 5, 27) that is illustrated in Fig. 14(b), which displays the isosurface from Fig. 14(a) in the volume of model complexities. Here, the solution is found at the point where the isosurface of constant complexity and the desired residual level isosurfaces tangent each other. The result is reasonably close to the ground truth and thus indicates that the model selection was successfully performed.

The mean vesicle ${\mathbf{g}}_0^{\prime }$ and basis components in ${\tilde{\mathbf{G}}}^{\prime }$ of the selected model are illustrated in Fig. 15. One can clearly see the dominance of the radial components against the angular ones, whereas some asymmetric structures are characterised by the latter components. The importance of the radial components can be likewise seen in Fig. 14(b), as many radial and few angular components yields a small $\tilde{K}$, whereas a model with many angular and few radial components yields a much larger $\tilde{K}$. In Fig. 14(a), one can also see that the parameter $\tilde{K}$ is saturated to the value I₁ I₂ when both I₁ and I₂ are small but the surface bends as soon as there is redundancy in the combined bases elements that the second layer PCA is able to remove.

Fig. 15.

The selected HOSVD model $({\tilde{I}}_1,{\tilde{I}}_2,\tilde{K})=(6,5,27)$. (Top left) The mean vesicle ${\mathbf{g}}_0^{\prime }$; (from the second in top left to right) the 27 basis images in the order of importance.

To understand how the vesicle removal affects the protein signal and how the vesicle removal performs under varying amounts of noise, we first generated simulated protein projections [38] and added Gaussian noise from increasing inverse signal-to-noise-ratio (SNR). A realistic amount of noise versus the membrane protein is about ten times in the inverse SNR [38]. Then we projected the simulated images, without added vesicles, onto the above trained vesicle subspace and onto its orthogonal complement. Due to the assumption of additive Gaussian noise and linearity of the subspace projection, it is equivalent to project the noise and protein components separately and superimpose the results, as Fig. 16 illustrates. It can also be seen what part of the protein signal and noise signal disappears into the vesicle subspace. From the simulated protein particle, 5% is projected onto the vesicle subspace in the square L₂ norm that illustrates the overlap of the protein and vesicle subspaces. To see the effect of noise onto the protein, we simulated the relative energy of the artifact caused by the vesicle removal as reported in Fig 17. Similarly, we evaluated the effect of noise on the removal of simulated vesicles [40]. It can be seen that the noise completely dominates the projections when the inverse SNR grows to hundred as the norm ratios tend to unity.

Fig. 16.

Illustration of how the projection onto the vesicle subspace (bottom row) and to its orthogonal complement (top row) affects the simulated protein signal (middle row, left), noise (middle row, middle) and the protein signal superimposed with noise (middle row, right). The amount of noise is controlled by the parameter α. Both the protein and noise have non-zero projection onto the vesicle subspace that induces the artifacts shown on the top row.

Fig. 17.

Relative artifact energy in the function of the inverse signal-to-noise-ratio. (Solid line) Induced relative artifact defined as the square L₂ norm ratio between the projection of protein plus noise onto the orthogonal complement of the vesicles minus the true protein signal and the projection onto the orthogonal complement. (Dashed line) Induced relative artifact in simulated vesicle removal defined as the square L₂ norm ratio between the projection of the simulated vesicle plus noise onto the vesicle subspace minus the true vesicle and the projection onto the vesicle subspace.

Finally, we compared the proposed approach against the robust principal component analysis (RPCA) [6]. However, in our application, there are three distinct components opposed to the standard setting of the RPCA since we have the low rank component (vesicle), sparse component (protein and background dirt), and noise which is not sparse. The original RPCA problem considers

$$\underset{\mathbf{L},\mathbf{S}}{\text{min}}\parallel \mathbf{L}{\parallel }_{\ast }+\lambda \parallel \mathbf{S}{\parallel }_1:\mathbf{L}+\mathbf{S}=\mathbf{M}$$ (39)

where L is a low rank matrix, S is the sparse part of the measurement matrix M = A₍₃₎ − A_0,(3), and λ is a parameter. To adapt the RPCA to our problem we thus modified the RPCA problem to

$$\text{min}\parallel \mathbf{L}{\parallel }_{\ast }+\lambda \parallel \mathbf{S}{\parallel }_1:\parallel \mathbf{L}+\mathbf{S}-\mathbf{M}{\parallel }_{\text{fro}}=C_{\sigma },$$ (40)

where C_σ is a constant fixing the noise level, and solved it by Alternating Direction Method of Multipliers (ADMM) [4].

The resulting modified RPCA components are shown in Fig. 18. It can be seen that the low rank components are more noisy or less smooth than the basis elements of the proposed method (Fig. 15). It is natural since the modified RPCA model is not generated by separating the radial and angular parts that significantly reduces the dimensionality of the proposed method. The modified RPCA model likewise fails to capture the asymmetries of the vesicles. The vesicle removal results, using the learnt low rank components for a novel micrograph, are qualitatively compared in Fig. 19 and Fig. 20. The modified RPCA model works reasonably well, but the vesicle removal leaves more artifacts in the micrograph than by the proposed method. It can be seen (Fig. 20) that the noise-free estimates of the vesicles, displayed in the center column, capture the individual structural features, yet are robust to structural noise both inside and outside the vesicles (row 3, 4), or attached protein (row 2). The vesicle removal by the two methods using a complete micrograph is furthermore compared in Fig. 21, where it can be seen that the modified RPCA model consequently leaves vesicle artifacts. We conclude that the modified RPCA method leaves more artifacts after removal than the proposed method. The latter is also robust to the global illumination changes over the micrographs, as the vesicle structures have been successfully removed without an addition of notable systematic artifacts.

Fig. 18.

Modified RPCA model diving the vesicles into low rank, sparse and noise components. (top) 14 of the most significant of the 78 low rank components; (middle) 14 sparse components; (bottom) 14 dense, noise components. By this decomposition the vesicle image naturally divides into the vesicle, substructure, and noise parts, respectively.

Fig. 19.

Vesicle image removal for novel vesicles by using the modified RPCA model with 78 low rank components. (Column a, b) normalised vesicle images; (c) projection onto the vesicle subspace; (d-e) the removal result. The images (b-d) are slightly blurred by a Gaussian filter to aid visual inspection, c.f. Fig. 20.

Fig. 20.

Vesicle image removal for novel vesicles by using the proposed method with the automatically selected model $({\tilde{I}}_1,{\tilde{I}}_2,\tilde{K})$ = (6, 5, 27). (Column a, b) normalised vesicle images; (c) projection onto the vesicle subspace; (d-e) the removal result. The images (b-d) are slightly blurred by a Gaussian filter to aid visual inspection, c.f. Fig. 19.

VII. Conclusions and Future Work

In this paper, we have proposed a method to learn and remove objects from images, where our application is the removal of lipid vesicle structures from electron micrographs to facilitate 3d reconstruction of proteins. To model the object, we first estimated the vesicle model subspace by the Higher Order Singular Value Decomposition in the polar coordinate plane. The two dimensional HOSVD basis was then converted to one dimensional by proposing a PCA layer to the HOSVD model. The model selection was performed by applying Occam razor and Morozov’s discrepancy principle for the hierarchical model. Experiments showed that the proposed approach efficiently captures the vesicle structure variations in a low dimensional subspace and therefore the removal procedure has a minimal impact on the underlying signals such as those of the interesting membrane proteins. In the future [18], we will apply the vesicle removal method to facilitate the membrane protein reconstruction in cryogenic single particle electron microscopy, where the vesicle structures deteriorate the protein signal. It is likewise possible to integrate the vesicle model directly into the reconstruction problem that we deem as a promising approach.

VIII. Software

The computer source code will become available at authors’ web page.

Biographies

Katrine Hommelhoff Jensen received the M.Sc. degree from the Department of Computer Science, University of Copenhagen, Denmark, in 2006, with a specialization in image analysis and 3d computer graphics. She is currently pursuing the Ph.D. degree with the Image Section, Department of Computer Science, University of Copenhagen. She was a 3d Software Developer at the company 3Shape A/S, Denmark, from 2004 to 2007, and at the company Follow-Me! Technology Systems GmbH, Germany, from 2007 to 2011. Her work is focused on statistical methods for single-particle cryo-EM. Her research interests include bio-/medical image analysis and computer graphics, in particular 3d imaging and reconstruction, shape analysis, mathematical modeling, and statistical inversion.

Fred J. Sigworth was born in Berkeley, CA, in 1951. He received the B.S. degree in applied physics from the California Institute of Technology, Pasedena, in 1974, and the Ph.D. degree in physiology from Yale University, New Haven, CT, in 1979. He was a Research Fellow with the Max Planck Institute for Biophysical Chemistry, Göttingen, Germany, from 1979 to 1984, where he was with E. Neher on the development and applications of the patch-clamp technique for recording single ion-channel currents. Since 1984, he has been a Faculty Member with the Department of Cellular and Molecular Physiology, Yale University. His research interests include the structure and function of ion channel proteins and the development of techniques to study ion channels. He is a member of the Biophysical Society, the Society of Neuroscience, and the American Scientific Affiliation.

Sami Sebastian Brandt received the Ph.D. degree from the Helsinki University of Technology, Finland, in 2002, and the habilitation degree on the geometric branch of computer vision from the University of Oulu, Finland, in 2007. After the Ph.D. degree, he spent one year as a Research Scientist with the Instrumentarium Corporation Imaging Division, Finland, a couple of years in the Helsinki University of Technology, Oulu University, Finland, Malmö University, Sweden, and Nordic Bioscience Imaging/Synarc Imaging Technologies in Denmark. He is currently an Associate Professor with the Image Group, University of Copenhagen, Denmark, and a Senior Mathematical Software Developer with 3Shape. His research interests include applied mathematics, statistical inverse problems, Bayes methods, electron tomography, single particle reconstruction, geometric computer vision, and image analysis.