What is in the black box? – A perspective on software in cryoelectron microscopy

Abstract

This article bemoans the demise of truly modular open-source image processing systems, such as SPIDER, in recent years’ development of tools for three-dimensional reconstruction in cryo-electron microscopy. Instead, today’s users have to rely on the functionality of software systems that have little or no transparency. As a consequence, users of such packages no longer gain a conceptual understanding and intuitive grasp of the analytical routes leading from the stream of input data to the final density map. Possible remedies of this situation with free software are discussed.

Significance

This Perspective addresses an important aspect of science education. Cryoelectron microscopy has become one of the pillars of structural biology, but the data processing has become quite complex, increasingly organized in “black boxes” no longer transparent for the user. Earlier, modular image processing systems had the advantage of providing a conceptual training ground in signal processing for the novice. We conducted a careful review of several software systems and singled out three as a remedy.

Main text

“In science, computing, and engineering, a black box is a device, system, or object which can be viewed in terms of its inputs and outputs… without any knowledge of its internal workings” (1). Closely related, semantically, is the polemic phrase often used to characterize uncritical application of software and other tools in science: “garbage in, garbage out.” Given this concept, and the blemish associated with it, asking the question “What is in the Black Box” may be viewed as a violation of basic etiquette, a question only posed to provoke. But my question is rather meant as a lead to recall a time when software was transparent, and as an appeal to help us think of a remedy.

The case in point is the software used for image processing in electron microscopy. By now, the power of single-particle cryogenic electron microscopy (cryo-EM) in determining high-resolution structures of molecules in solution is widely appreciated. Resolutions in the range of 3–4 Å are almost routinely obtained, and even staggering resolutions up to 1.2 Å have been recently reported (2,3). After the new single-electron detecting cameras were commercially introduced, in 2012, cryo-EM has contributed in major ways to the fight against several pandemics, including the most menacing one, coronavirus disease 2019 (COVID-19) (4). Ignoring for the moment the extra challenge posed by situations where a diversity of structures coexist in the same sample, the task for the image processing software is that it has to enable the computer to take in a flood of two-dimensional (2D) images that are projections of different “copies” of a molecule in many different views, and in the end crank out a three-dimensional (3D) image, or Coulomb potential map, that is consistent with the observed data. A realistic example, taken from our area of biological focus, is a 3-Å density map of the ribosome containing hundreds of thousands unknowns that is reconstructed from 50,000 low-dose EM images, each presented as a 300 × 300 array, or a total of 4.5 × 10⁹ extremely noisy measurements. Even for a single structure, the problem is not well defined and has a huge number of solutions that are all compatible with the input. It is, in the lingo of mathematics, ill-conditioned. The black-box software has dials to turn, and, depending on the settings, one of the many solutions is selected. Even though the user can make several choices that affect the outcome, this falls short of having true control because the workings inside the box, and the way the dials affect the outcome, are hidden. The only saving grace is the fact that many of the solutions are so similar as to be virtually indistinguishable. Still, without a true understanding of the concepts the user’s parameter choices may be off, and nonoptimal or nonsensical results may be obtained by a newcomer.

Old-timers will recall that the field started out quite differently, with software that was modular with precisely defined functionality of each module. Perhaps a good snapshot in time, already at an advanced stage, is a special issue of the Journal of Structural Biology co-edited by Ueli Aebi, Bridget Carragher, and Ross Smith in 1996 (5). In a table accompanying the editorial introduction, 10 general packages were listed that shared similar coverage of the whole canon of necessary operations in EM-related image processing: EM, Eos, I.C.E., IMAGIC, MDPP, PIC-III, Semper, SPIDER/Web, and SUPRIM. Most of these had the fine-grained script-controlled modularity that lends itself to both the exploration of algorithms and the training in concepts. The majority was software that was in a sense an open book; it allowed user insight into the underlying code as well as script-based control of the computing workflow in a transparent way. As each module was designed to perform a standard operation, the software could be readily used, through building a workflow from the modules, to teach, illuminate, and explore the mathematical concepts underlying the processing strategy. Quite importantly, it allowed participation of sufficiently trained users in the development of new features that might be required to address problems in their research that had not been encountered before.

Here, we do not wish to indulge in nostalgia because, as a practical matter, much of the recent progress has been achieved thanks to the development of highly specialized packages making use of parallel computing and, increasingly, machine learning; rather, we want to give a bit of history to point out what has been lost in the transition, and add some thoughts on how the trend could be reversed.

SPIDER (6, 7, 8, 9), the script-controlled modular image processing system developed for electron microscopy at the end of the 1970s, received its inspirations from VICAR, also of modular design. VICAR, which stands for Video Image Communication And Retrieval, was a software system developed in the mid-1960s in the Jet Propulsion Laboratory (JPL) for processing images set from unmanned interplanetary space missions. During an extended visit at the JPL in 1971, one of the authors (J.F.) docked his own EM-related Fortran programs into VICAR to benefit from that system’s infrastructure, such as the support of command scripts with do-loops and conditional jumps. At the time, anything resembling today’s workstations incorporating a high-resolution graphics screen did not yet exist, so all computations were started as batches from a console, and resulting images could not be viewed until later, after being recorded on film by a special device.

As soon as high-resolution-graphics-supported workstations appeared, in the mid-1980s, interactive image processing became feasible. The idea behind the design of SPIDER was that images could be processed right in front of you in an interactive session driven by commands. After a prompt (operation:), the user enters a command (e.g., RT, for “rotate”), upon which another prompt asks for 1) the name of the image file to be rotated, 2) under which name the output file is to be created, and 3) the rotation angle. After the operation has been executed, the next operation prompt appears. In the most useful configuration—from the training point of view—the input image and output image are successively shown on the screen of the workstation along with the script dialog text. When the first graphics workstations became available, the Frank team actually created a version of SPIDER in which every operation to write an output image onto disk was immediately echoed by a write onto the screen. Thus, the effect of each operation was readily apparent, and a misstep in the choice of a parameter value was immediately recognized. In the above example, we chose the rotation operation for its simplicity; but in reality, there were in a range of 400 operations covering the wide range of 2D signal processing, among these many operations in Fourier space such as convolutions and correlation functions, as well as linear and nonlinear contrast enhancement functions. All of these could be invoked sequentially, accompanied by a display of the result after every single step. In this way, the interactive session could become a self-guided tutorial. In addition, the record of the interactive session, accumulated in a file, could be subsequently used to run the entire session in a batch-processing mode.

Only a certain minimum of mathematical concepts in signal processing are actually required to understand basic concepts underlying EM-related image processing, such as sampling, aliasing, low-pass and high-pass filtering, cross-correlation, convolution, projection, and reconstruction via weighted back-projection. Essentially one has to grasp the meaning of a Fourier series expansion in one, two, and three dimensions. Next one has to develop an intuition for the duality of operations in real space and Fourier space: convolution in one space becomes scalar multiplication in the other, and vice-versa. That intuition can be fostered by playing with the operations in one space—sliding variables and parameters, for instance—and watching what happens in the other space. Quaint atlases of “optical transforms” (10,11) come to life right in front of you (Fig. 1). SPIDER and some modular image processing systems developed later on provided such a “sandbox” playground for the first generations of students in the new field. Upon being contacted for this article, some former postdocs that were in the Frank laboratory during the formative period of single-particle techniques (before 2000) professed missing this aspect in the software used today.

Figure 1.

Examples of shape transforms (“optical transforms”). The right-hand panel shows various shapes, and the left-hand panel shows the corresponding Fourier transforms obtained either with the aid of an optical diffractometer or a computer (Reproduced, with permission, from Harburn et al. (10)).

We will give an example for the way the building blocks of such a modular system could be readily put together into a workflow to achieve a defined goal. The example is the 2D alignment of two images that have a common motif, as happens when two copies of a molecule show the same view in the electron microscope but appear in arbitrary in-plane orientations and x-y positions, and with different realizations of noise. Such 2D alignment is needed for forming an average of the common signal or for any other operation, for instance classification, in which images are being related to one another. The flow diagram (Fig. 2) represents an implementation of a reference-free 2D alignment procedure proposed by Langer and co-workers (12). The algorithm makes use of the fact that the autocorrelation function (ACF) is shift-invariant, so it can be used to find the orientation of two images, no matter where they sit in their reference frames. This is accomplished by computing the angular cross-correlation function of the two ACFs, which is maximized for a rotation angle θ that leads to a perfect overlap of the two ACFs. After the second image is rotated by –θ, the amount of shift required to bring the common motif into exact overlap is obtained by computing the 2D translational cross-correlation function. As explained in the legend, the latter step needs to be repeated using the angle –θ +180° because the ACFs of an image and its 180 rotated version are identical. The decision between which angle is correct is made by comparing the values of the translational cross-correlation function for –θ and –θ +180°).

Figure 2.

Flow diagram of reference-free alignment of two images realized in SPIDER using the following list of operations: float (i.e., pad into an image twice the size of the original), autocorrelation, angular cross-correlation of two ACFs, rotation, cross-correlation, peak-search, shift. Not shown in the diagram is an additional computational decision, made by trying both θ and θ + 180, on which of these two possible solutions is correct, as the ACF is centrosymmetric. (Redrawn from (13)).

We hope in the foregoing we have been able to convey the message that something important has been almost completely lost in the quest for speed, high throughput, and perfection. Fortunately, on the other hand, general software platforms with great versatility such as MATLAB (The MathWorks, Natick, MA) have the functionality required to pursue the kind of conceptual explorations we have in mind. The difference is that previously the “sandbox” was part and parcel of the software system used for image processing, and readily available, whereas nowadays, a sufficiently motivated novice has to overcome an extra hurdle to learn how to use an additional tool set. We would argue that this extra obstacle is not trivial for users without math background. SPARX (14), a newer package that did not exist at the time of the 1996 special software issue of Journal of Structural Biology, does contain many of the required pieces, but what is missing is a roadmap on how to generate a convenient training tool.

Following this train of thought, we have undertaken an exploration of eight software systems/programming languages for suitability as training and exploration environment of the kind discussed, and have summarized the features in tabular form (Table S1). As a result of the comparison in terms of some relevant criteria (interpreted versus compiled language, static versus dynamic type checking, object-oriented programming (yes/no), and multiple dispatch (yes/no), “included libraries,” signal/image processing toolboxes (yes/no), data science (availability of statistical and machine-learning packages) capability (yes/no), good documentation/tutorials (yes/no), and “run-time performance”), we lean toward recommending three platforms, MATLAB, Python, and Julia. However, only the latter two are open source. Here are the reasons for this preference.

MATLAB is a high-level, multiparadigm and high-performance scientific programming language (https://www.mathworks.com/help/matlab/index.html). It combines advanced graphics, visualization, and technical computing in an integrated, user-friendly, and interactive environment which helps in testing and learning mathematical theories and rapid implementation of algorithms.

Python is a multiparadigm, general-purpose, high-level programming language with easy to read and clear syntax (https://docs.python.org/3/). It is the most popular open-source language at present. Fortunately, scripts of a tutorial for math underlying single-particle cryo-EM processing are already available in a github repository (15). These include notebooks on numerous subtopics on the physics of the electron microscope and image formation (electron optics, electron diffraction patterns, contrast-transfer function), on mathematical concepts (Fourier transforms, convolution, coordinate systems and rotations, interpolations, Fourier slice theorem, 3D reconstruction), on Bayesian inference and estimation theory (maximal likelihood, 2D and 3D image classification, covariance), and on some important image processing tools (alignment via cross-correlation, 3D reconstruction via weighted back-projection, resolution estimation via Fourier shell correlation, detector noise modeling). There are simple object models to experiment with and learn about the basics of phase contrast image formation and data processing.

Julia is a relatively new, high-level, multiparadigm, fast, dynamic programming language with a mathematics-friendly syntax and is geared toward high-performance numeric computation (https://julialang.org/). Some of the language design and features of Julia make it a notable alternative to MATLAB and even Python. However, no cryo-EM image processing tutorial using Julia is available yet.

For details on the features of all eight software systems/program languages reviewed we refer to the annotations of Table S1.

Finally, as training curricula are drawn up in resource centers for cryo-EM, we would suggest it would be a good idea to include a section dedicated to the training in basic signal procession concepts, and introduce users to software, preferentially open source, that lends itself to trial-and-error explorations with experimental or simulated ground-truth images.

Anything, in other words, that makes the box a bit more transparent.

Author contributions

J.F. conceived this perspective. S.M. researched the software solutions. J.F. and S.M. jointly wrote the manuscript.

Editor: Meyer Jackson.

Supporting citations

References (15–34) appear in the supporting material.