Perspectives on structural molecular biology visualization: from past to present

Abstract

Visualization has been a key technology in the progress of structural molecular biology for as long as the field has existed. This perspective describes the nature of the visualization process in structural studies, how it has evolved over the years, and its relationship to the changes in technology that have supported and driven it. It focuses on how technical advances have changed the way we look at and interact with molecular structure, and how structural biology has fostered and challenged that technology.

Introduction

Visualization has been an enabling technology in structural molecular biology for as long as the field has existed. The process of visualization, itself, implies human involvement since its ultimate impact is insight. In general, visualization can aid in three functions that are essential components of the scientific enterprise and fundamental to our understanding of the “unseen” world of structural molecular biology, synthesis, analysis and communication:

• Synthesis involves the process of creating a model to embody or integrate experimental observation or to develop and encompass a structural hypothesis.

• Analysis entails the examination and exploration of either data or models to explain observed phenomena, derive new hypotheses, or suggest new experiments.

• Communication involves the imparting or sharing of information and knowledge about the systems or methods under study.

These activities pre-date the advent of computer technology, as do the use of visualization tools in their service. Moreover, the evolution of molecular graphics and the visualization of biomolecular structure demonstrate a significant symbiosis between science and the technology that enables it. There are many good reviews of the history of molecular graphics and illustration ^{2; 3; 4} We focus here on the how technical advances have changed the way we look at and interact with molecular structure. This perspective is not meant to be exhaustive, but rather one person’s perspective on a continually evolving field.

Looking back on the development of what originally was termed “molecular graphics” we see a history of invention and innovation driven by a need to comprehend the complex nature of the molecular world. Graphics and visualization tools have been critical in enabling and catalyzing the remarkable progress of structural molecular biology from the past half century to today. Conversely over this time, the field of structural biology has helped to drive development and commercialization of computer graphics and visualization technologies.

The Past

Origins

Physical models were the earliest interactive three dimensional molecular visualization tools. Examples such as the CPK (Corey Pauling Koltun)⁵ and Dreiding⁶ models built upon earlier work going back to the mid-19^th century, when chemists such as Hoffmann and Kekule² built wooden ball-and-stick or wire models demonstrating element combining ratios or bond connectivity of simple chemical compounds. As X-ray crystallography began to produce three-dimensional molecular structures during the first half of last century, the need to represent them in accurate and comprehensible form increased the use of these physical models. In the 1950s they began to be used to model biological molecules. Pauling, who helped develop the CPK model and adopted the use of physical models throughout his career, used the simple process of folding a paper with a drawing of a polypeptide chain, coupled with his knowledge of the hydrogen bonding potential of amino acids to deduce the alpha helix as a folding motif for proteins⁷. Likewise, Watson and Crick, playing with brass models of nucleotide base structure, were able to explore geometric combinations to explain Franklin’s fiber diffraction data and synthesize a model of the structure of DNA that revealed its genetic function. In a real sense both examples demonstrate how physical visualizations can promote exploratory interaction and serve as “analogue computers” leading to new insights^8;9.

In the 1960’s computer technology became a critical catalyst driving progress in crystallography and structural molecular biology. Initially the computational power was dedicated to deducing electron density maps from the x-ray diffraction data. Typically these maps were displayed using computer line printers to produce pages of two-dimensional arrays of density values ranging from 0 to 9. Each page was hand-contoured by isovalue and transferred onto balsa wood sheets which were cut out and stacked to make a physical representation of the electron density ^{10; 11} (see Figure 1). As computer-driven X-Y pen plotters became available the electron density contours could be drawn out automatically on transparent plastic sheets that could be stacked with appropriate spacers to give a direct 3D spatial view of the electron density map. Also at that time pen-plotters were being used to draw small molecule structures. The ORTEP (Oak Ridge Thermal Ellipsoid Plot) program by Carrol Johnson ^{12; 13} was the most widely used and cited computer program of its day. It enabled the accurate plotting of stereoscopic molecular structures and introduced the concept of thermal elipsoids, which visualized the anisotropic temperature factors of each atom in the small molecule structure. However, the creation of 3D molecular visualizations of protein structures from electron density maps relied upon the building of physical models, which was tedious and time-consuming. An ingenious apparatus devised by Fred Richards at Yale, utilized stacked transparent sheets with plotted electron density contours and brass Kendrew model parts to be screwed together to build polypeptide structure¹⁴. Model support rods were placed in front of a large half-silvered mirror and with the appropriate lighting the Kendrew parts, as seen in the mirror could be built into the electron density as seen through the mirror (Figure 2). Once the physical model was built, the 3D coordinates of each atom in the model had to be measured and recorded (X-Y by projection of sight line, and z with a plumb-bob).

Figure 1.

Balsa wood model of hemoglobin from Perutz Lab. ca.1960.

Figure 2.

Stacked electron density (left) and brass Kendrew protein model (right) in Richards Box. ca. 1975

In the mid-1960’s, using a one-off oscilloscope-like line drawing system at MIT, Cyrus Levinthal and Robert Langridge were able to display and rotate models of lysozyme and myoglobin created from their known 3D coordinates¹⁵. This event marked the beginning of interactive computer-generated molecular graphics. It took several years before the technology matured to a commercial stage, but from those early applications, interactive molecular graphics was seen as a driving application for the technology.

As new protein crystal structures were solved, the opportunity to communicate structural molecular biology opened up. Probably the most influential book on protein structure from that decade was “The Structure and Action of Proteins” by Dickerson and Geis¹⁶, published in 1969. The artistic input that illustrator Irving Geis used to represent molecular structure and protein folds in that book and in a Scientific American article¹⁷ brought protein structures to life and made them accessible to a broad audience. Geis’s drawings also served as an inspiration for the now ubiquitous ribbon diagrams of protein topology made popular by Jane Richardson in the following decade ¹⁸. The evolution of such molecular illustration work is covered by Goodsell and Jenkins [This issue]. It should be noted that much of what was originally developed for illustration and communication purposes, has been incorporated in scientific molecular visualization tools. Initially there was a strong distinction between “working graphics,” where rapid feedback and high interactivity are paramount, and “publication graphics” where clarity of presentation is key. As technology has improved over the decades, the distinction between these modes has blurred somewhat.

In the early 1970s interactive monochrome CRT vector displays became commercially available. Several small companies such as Vector General, Evans&Sutherland, and Adage started up to produce and market these devices, initially for military and aviation applications. These monochrome displays used move and draw operations on an electron beam to create lines and dots on a phosphor screen at interactive frame rates. Remarkably the beam could address the equivalent of 8000×8000 pixels - a resolution that is only now being approached in color 8K displays. Depth cueing (dimming lines farther away) and interactive rotation enhanced three-dimensional perception. Meanwhile, with the advent of more powerful and available computers, the field of biomolecular crystallography expanded, and by the mid 1970s several groups began to develop software to move the protein model-building task from the physical Richards Box onto interactive computer displays. Programs, at the time termed “Electronic Richards Boxes,” such as Bilder¹⁹, Frodo²⁰, MMSX²¹ and Grip²² were developed by crystallographers and computer scientists to interactively build polypeptide structures into the 3D computed iso-contoured electron density maps on the CRT display (Figure 3). These codes were able to speed up the process of protein model building several fold and, in addition, automatically produce more reliably captured atomic coordinates. The commercial availability of the display technology along with the purpose-built electron density fitting programs brought more scientists into the field of protein crystallography and grew the demand for interactive computer display devices. Thus, technical advances in display hardware and molecular software not only greatly facilitated rapid growth and progress in structural molecular biology, but also provided an expanding market for those early and very expensive graphics devices. In fact molecular graphics was a “killer app” that helped sustain and build the nascent computer graphics industry²³. By the early 1980s there were close to 100 labs around the world that had purchased interactive display systems for biomolecular graphics²⁴.

Figure 3.

Screenshot of Bilder ca. 1977

Evolution

The 1980s saw an explosion of applications and innovation in structural biology and molecular graphics. The enhanced ability to build 3D biomolecular models on the computer drove new applications for the synthesis, analysis and communication of these previously unimagined protein structures. One innovative and very useful structure-based analytical representation developed during this time was the solvent-based molecular surface, originally conceived by Lee and Richards in 1973 ²⁵ This surface describes the interface between a protein’s atomic structure with its surrounding water solvent, and gives a quantitative measure of effective surface area for these structures and their interactions. Michael Connolly then at UC Berkeley/UCSF, developed the first program to compute and display the “solvent excluded” molecular surface²⁶. With the line-drawing graphics systems available at the time, the surface was visualized by placing dots on it. This representation had the benefit of allowing the underlying bond structure to be seen in the context of the surface, and became a standard molecular visualization technique (Figure 4). Also during this time, molecular dynamics trajectories of a protein’s motion added the time dimension to the static view of protein structure. While these simulations took hours or days to compute, the resulting protein trajectories could be re-played and viewed interactively.

Figure 4.

Dot Molecular Surface with protein backbone trace ca. 1983

With the fields of structural biology and computer graphics co-evolving over this period, the representation and communication of biomolecular structures and their function developed rapidly. Color line-drawing displays became available in the early 1980s and were quickly adopted for molecular graphics. Color added an extra visual dimension and enabled clear visual discrimination of atom-types, chain identities and multiple molecules in complexes. During this time color raster displays became more widely available. These devices utilized a frame buffer to build up a picture using a raster scan-line like the television, and an early use was the display of shaded surfaces of large molecular structures²⁷. The resolution of the available color displays was as high as 512×512 pixels (Figure 5). However for molecular graphics the color raster displays were used primarily to produce still images for presentation since interactive rotation and display of a protein with hundreds to thousands of atoms represented as spheres or surfaces was beyond computational capabilities at the time. Raster-based graphics enabled more “realistic” molecular renderings by simulating the effects of light on solid surfaces. Shaded and highlighted surfaces provided additional visual cues about shape, and could approach the appearance of plastic CPK physical models²⁸.

Figure 5.

Image from AED 512 raster graphics display of protein-protein interaction between trypsin and trypsin inhibitor, ca. 1984

Initially molecular graphics programs were developed for specific functions, such as electron density fitting or display of dynamics trajectories. As an increasing number of protein structures became available, more general molecular graphics programs were developed that enabled exploration, analysis and communication. Three of the earliest general purpose programs were MIDAS²⁹ HYDRA ³⁰ and GRAMPS/GRANNY^31;32. These programs went beyond electron density fitting to provide general graphics functionality for examining molecular structure and properties. GRAMPS was developed as an interpreted graphics language that had interactive features built into it that enabled the composition of animations. It was used to produce a number of molecular movies throughout the 1980s that detailed the structures of viruses, antibodies, and enzymes. These 16mm films were shot from the computer display onto 16mm movie film with an animation camera triggered by a solenoid to advance each frame. In one instance, GRAMPS was used to produce animated segments for an OmniMAX film shown at Disney’s EPCOT Center³³. The molecular segments depicted traveling along the major groove of DNA (Figure 6) and the assembly of viral subunits as viewed from inside. The segment was produced for large format IMAX film (equivalent of 3 frames of 70mm movie film) projected onto the OminMax spherical dome. This meant that the line-drawn images had to be computationally distorted³⁴. They were then written out frame-by-frame onto 6250bpi magnetic tapes and sent to a film recorder. The two-minute segment required over 60 nine-inch diameter tapes (roughly 6GB).

Figure 6.

Frame from Omnimax animation of trip down DNA major groove, for Disney’s EPCOT Center, 1983

Over the decade of the 1980s the remarkable evolution of computer hardware - the advent of microprocessors, very large scale integration (VLSI), personal computers, and scientific workstations - increased the accessibility of molecular graphics. Initially structural researchers themselves wrote the molecular modeling software. By the middle of the decade, the demand was such that several commercial companies such as Biosym, Molecular Design and Tripos had been established to market molecular graphics and modeling software. By the end of the decade structural scientists in academic and industrial research settings had a wide variety of use-tested hardware and software platforms with which to do molecular modeling and visualization.

Also during these early decades computer graphics became a separate field of study within the discipline of computer science. The Association for Computer Machinery (ACM) initiated a special interest group devoted to computer graphics and interaction (SIGGRAPH) which had its first Annual Conference in 1974 with 600 people in attendance³⁵. In the 1990’s SIGGRAPH Conferences had up to 45,000 attendees. During this decade the field of visualization, itself, became a distinct discipline, focusing more on the development of methods to apply in scientific visualization, information visualization and visual analytics. Over this time novel graphics and interactive technologies were developed with a number of them driven by biological problems. New interactively capable raster-based displays supplanted the line drawing displays, and graphics algorithms focused on photorealism, geometric modeling, volume rendering, and texture mapping. Along with the more powerful graphics systems, new scientific visualization environments such as the Application Visualization System (AVS)³⁶ were developed and marketed that used a graphical programing paradigm with data flow networks of computational modules to enable rapid implementation of custom scientific visualization applications.

The 1990s saw the rapid growth in the number of protein structures deposited in the protein data bank (PDB). By the end of the decade it contained over 10,000 structures. The development of programs such as Molscript³⁷ enabled the creation of illustrative shaded drawings of protein structures, including ribbon diagrams. Rasmol³⁸ created an easy and accessible user interface to interactively view biomolecules on raster devices. Importantly, this decade saw the wide-use of the Internet, which made the ever-growing number protein structures more accessible and fostered the creation of web-based browser molecular graphics, such as Chime and Jmol³⁹ New modes of visualization and interaction were introduced and tested on molecular applications, including force-feedback haptic displays⁴⁰ data sonification, and virtual and augmented reality⁴¹. As with much of technology, the time between invention and broad utilization can take many years or even decades. One problem has been the traditional communication barriers between the computer science and scientific communities. Too often researchers in computer science develop tools and algorithms no real application in mind, while scientists view the details of their application as paramount and treat computer scientists as simply programmers. There are few, if any, conferences that are equally attended by both communities and publications and proceedings are rarely shared. In addition, not all tools and techniques fit the needs of a particular scientific discipline. Thus acceptance and usage of a given graphics technology or algorithm for a particular application domain, such as molecular visualization, is never guaranteed.

The Present

In the intervening years, over 130,000 protein structures have been deposited in the Protein Data Bank (http://www.rcsb.org/stats/summary) and we now have high definition color graphics displays in our pockets. Utilizing a growing arsenal of biophysical techniques such as electron microscopy and fluorescence light microscopy, we are generating, at accelerating rates, bio-structural data from experimental approaches that span from atoms to cells. Computational power and Graphics Processing Units (GPUs) now enable real time interactive display of billions of atoms, millions of proteins. GPUs are now considered “a sixth computer technology revolution”⁴². Currently available high-end GPUs have upwards of 5,000 processors and 24 GB of shared memory. While GPUs were originally developed for rendering images for 2D display devices with a mapping of pixels to processors, their utility is applicable to any process that can be broken down into many independent processes that can run in parallel. Today, applications that range from molecular dynamics to autonomous vehicles utilize the power of these processors. How have these advances in science and technology affected the still evolving world of molecular visualization and interaction? How has the commodification of 3D interactive computer graphics, and its broad adoption in society changed the way we look at molecules?

While in the early days of computer-based molecular graphics crystallographers and other molecular scientists developed their own software to visualize, explore and analyze the structures they were solving, today computer graphics and interactive techniques are being developed and utilized by a large number of different communities that range from scientific visualization to entertainment film and game production. Academic and commercial development has spread to fields ranging from computer graphics algorithm development, scientific and information visualization and from hardware design to the user interface, digital aesthetics, art, education and communication. The implication of this is that while billions of dollars are now spent on the technology and development of digital visualization and interaction, molecular graphics is no longer a principal driver. It has now become the task of the structural biology community to explore and adapt the technologies from these other fields to serve the needs and expand the capabilities of molecular graphics and visualization in synthesis, analysis and communication. On the other hand, since biology is so complex, its modeling, visualization and analysis continue to challenge the technology. Here we survey some of the current approaches and the utility of new developments in the service of our science.

Software Development and Integration

One major challenge facing the molecular graphics and visualization community is how to best take advantage of all of the existing visualization, interaction, and animation advances that have been developed for other purposes. These advances include highly capable modeling techniques, such as geometric meshing, Boolean operations on complex geometries, rendering techniques that include realistic or special lighting effects, texture and bump mapping on complex surfaces, and interaction and animation approaches that include collision detection, kinematics and other physically based phenomena. In general there are two different ways to face this challenge. One path is to utilize existing high-end software; the other is to re-implement the desired technology or feature into existing molecular modeling programs. The re-implementation approach involves replicating prior work, which can be difficult and costly. The algorithms themselves may be tricky to reconstruct, and the implementation can be labor intensive in terms of coding time and integration into the existing molecular modeling package. On the other hand, utilization of the existing software packages can incur significant expense, if the codes are commercial, and in any case, the existing codes do not contain any domain knowledge of molecular structure, representation, or analysis. A number of high-end commercial modeling, rendering and animation packages have been developed for film and other entertainment industries. Among the most widely used are Maya⁴³, Cinema4D⁴⁴, 3DStudio Max⁴⁵, FormZ⁴⁶, and Houdini⁴⁷ Some other highly capable graphics software packages exist, such as Blender⁴⁸ and Sketchup⁴⁹, which are either open source or free for non-commercial use. Similarly, so called “game engines” such as Unity⁵⁰ and Torque⁵¹ have been developed to facilitate development of gaming software for real-time interactions utilizing fast rendering, physical simulations and adaptable user interfaces. The issue with using these high-end packages for molecular visualization is that they do not encompass any specific domain knowledge. Once a molecular structure is imported into this type of high-end software, it leaves behind its molecular and chemical data and its uses within the software. Thus, this creates an issue of continual import and export of structures between the graphics environment and the molecular modeling environment. To overcome this obstacle several approaches have been developed. One approach is to build molecular “plug-in” toolkits to extend a specific high-end graphics program with molecular modeling capability. Toolkits such as Molecular Maya⁵² and BioBlender^53;54, fall into this category. These toolkits make it easy to utilize the rendering and animation capabilities of the host graphics program while maintaining the ability to use molecular data, select molecular entities, such as atoms, amino acid residues and polypeptide chains, and specify conventional molecular representations, such as CPK, ball and stick, and molecular surfaces. Such toolkits have been successfully used in generating “Hollywood quality” animations of molecular structures and mechanisms (see, for instance, the work of Janet Iwassa^55;56 (Figure 7). One major drawback of these toolkits is that they require a new implementation of the molecular visualizaiton capability for each specific high-end graphics environment. Typically these implementations do not have the full capabilities of mature and widely used molecular visualization environments such as Chimera⁵⁷, PyMOL⁵⁸, PMV⁵⁹, or VMD⁶⁰ A second approach to building upon the capabilities of existing high-end graphics program is to “embed” an existing molecular visualization package into the host graphics program. This becomes possible when the high- end host program contains within it a widely used interpreter programing language, such as Python, which happens to be the case for many of these codes, such as Maya, Cinema4D, 3DStudio Max and Blender. We have taken advantage of the fact that PMV (the Python Molecular Viewer) was developed in an object-oriented, component based Python environment, and that the molecular visualization components could be easily separated from the graphical user interface (GUI) of PMV. Thus with a simple open-source connector plug-in, an embedded Python Molecular Viewer (ePMV)⁶¹ can be inserted into any number of high-end graphics codes without having to re-implement the large existing code-base for the molecular visualization components.

Figure 7.

Frame from Maya Animation on clathrin mediated endocytosis¹. Image courtesy of Janet Iwasa, U. Utah, structural data from Thomas Kirchausen, Harvard Med.

Molecular representation and rendering

Structural molecular biology is now over 60 years old, and while the breadth and range of structural information has advanced remarkably over this time, the fundamental representations of molecular Structures has evolved into a relatively standard visual vocabulary. CPK and ball and stick atomic representations (with standard coloring) tend to be used for the details of atomic resolution structures; ribbon diagrams ⁶² are used to highlight the backbone-level secondary and tertiary structural aspects of proteins and polynucleotide structures. A variety of geometric surface representations, including solvent excluded molecular surfaces⁶³ and coarse molecular and “blobby” isosurfaces⁶⁴ are used to convey the overall shapes of biological molecules and their assemblies. Molecular properties, such as hydrophobicity, polarity, electrostatics and sequence conservation, are represented by color encoding onto the geometric representations. Volumetric properties such as electron density and electrostatic fields are typically represented either as spatial isosurfaces at selected property values, or using direct volume rendering techniques, where the voxel values of the property of interest are translated into the optical characteristics of opacity and color⁶⁵ or as electrostatic field lines surrounding the molecular surface^66;67.

While this representational vocabulary has remained fairly static, advances in structural science have extended its range of applicability, and graphics technology has improved on its rendering and interactivity in numerous ways. Experimental NMR constraints and molecular modeling techniques, such as molecular dynamics, docking, and homology modeling can produce large ensembles of biomolecular structures and complexes. Static superposition and representation of these ensembles presents a problem in visual complexity, which has been addressed in a number of approaches, typically by using “clouds” of structures⁶⁸, variable diameter “sausage” backbone tubes, or volumetric rendering based on spatial occupancy⁶⁹. Alternatively, using interactive graphics with sequential display of structural information or trajectories opens up the time domain for comparison and analysis of dynamical information. “Flip-book” comparisons of several individual superposed structures can give a clearer picture of differences.

As the size and complexity of molecular structures has grown, the need for graphic clarity has become even more important. Here, advances in computer graphics rendering technology have had significant impact. The push toward realism in computer-generated imagery for film and games has resulted in a number of advanced rendering techniques. Depth cueing, or aerial perspective, has long been used in molecular graphics to give static and dynamic images a sense of dimensionality. The rendering of shadows can be helpful in interpreting depth and the relationship of component structures to one another, but can sometimes produce cluttered or confusing images. More recently, especially with surface representations, the development and use of ambient occlusion (AO) rendering has advanced the perceptual interpretation of shape. True ambient occlusion calculates how exposed each point on a surface is to ambient lighting or, put another way, how much of the surrounding environment can be seen from each point on a surface. For a typical, complex molecular surface this can be a time consuming calculation, and cannot be updated in real-time. However an approximation, called screen space ambient occlusion (SSAO), using pixel-based depth information is independent of scene complexity and can work in real time for dynamic representations (Figure 8). The program QuteMol⁷⁰ was the first molecular modeling package to incorporate SSAO. Subsequently many popular packages have implemented this type of rendering. This and other lighting effects that work on the screen-space image rather than the geometry can take advantage of the computer’s built-in graphics processing units (GPUs) to speed up rendering and interactive display.

Figure 8.

Comparison of shaded image (Left) and shaded image with SSAO (Right) of GroEL-ES chaperonin complex. Image from PMV.

The speed of transferring, rendering and interacting with complex molecular systems becomes a significant issue when the number of elements (atoms, chains, molecules, assemblies) becomes very large. A ribosome contains over 100,000 atoms, viral capsids can contain over of a million atoms. Current structures in the Protein Data Bank now contain assemblies as large as the capsid core particle of HIV, with more than 1300 protein chains and over 2 million independent atomic coordinates. The new MMTF binary file format from the PDB enables the efficient storage and transfer of such large structures⁷¹. Very large numbers of atomic spheres represented as triangulated polygonal surfaces is a challenge for molecular modeling software. However, screen space rendering methods, termed texture-based or imposter rendering, developed in the late 1990’s⁷² for GPUs utilize camera-facing two-dimensional glyphs of shaded spheres or cylinders and pixel depth information for each atom or bond - greatly reducing the polygon count and enabling interactive display of very large molecular complexes ⁷³ Other molecular representations for large-scale structures, using volume rendering and iso-surfaces created by meta-balls, can also be rendered in real time using GPUs⁷⁴ Advanced GPU-based imposter rendering techniques using WebGL, such as NGL⁷⁵ and LiteMol⁷⁶, enable interactive viewing of these massive molecular structures in a web browser (Figure 9).

Figure 9.

Interactive NGL ribbon image of HIV-1 capsid core particle.

Integrative Modeling and Visualization

Today, experimental methods can characterize the structures and distributions of biomolecules and their complexes both in isolation and in the context of cellular environments. These data, combined with information from genomics, proteomics, and other systems-based methods are being integrated into mesoscale structural models of complex cellular environments down to molecular and atomic detail. These models can be used in simulations to explore the dynamic and emergent behavior of crowded and heterogeneous cellular environments, and to test hypotheses regarding assembly and mechanism of multi-scale cellular processes. Building, visualizing and interacting with such large structural models is a significant challenge in the field of integrative structural and computational biology⁷⁷. Over the past several years there have been a number efforts to build complete structural models of cellular environments at molecular detail. This type of work has typically focused on a particular portion of a cell for example E.coli cytoplasm⁷⁸, M. genitalium cytoplasm⁷⁹, bacterial division machinery⁸⁰, synaptic vesicles⁸¹, and an entire synaptic bouton⁸². Other efforts have endeavored to build software frameworks to enable the construction of cellular environments at molecular scale [eg. PackMol⁸³, MolTemplate⁸⁴, MegaMol⁸⁵]. CELLPack⁸⁶ is an advanced version of this class of software. It uses distance field grid to discretize and describe a cellular model in a volume using multiple modular algorithms to interoperate on the model. It combines these algorithms to integrate three different major localization modes - volumetric, surface, and procedural - into a unified model. CellPACK has been used to produce models of systems as large as a complete mycoplasma cell, with about 30,000 proteins, 200,000 lipids, and one million base pairs of DNA, comprising over 50 million atoms⁸⁷‘⁸⁸ (Figure 10). Work on the GPU visualization of this type of mesoscale cellular model has demonstrated the ability to interactively navigate scenes with over 10 billion atoms at 60 frames per second⁸⁹ which is roughly the number of protein atoms in an E. coli bacterial cell. The idea of working with and interacting with terabytes of data has driven the field of big data analysis and visualization. A recent report in the visualization literature describes the interactive visualization of a dark matter cosmological simulation of a 32 terabyte data set with over a trillion particles by using advanced GPU technology⁹⁰. This effort involved several levels of data pre-fetching and caching from disk to RAM to GPU memory and simultaneously utilized two GPUs, one for visualizing the entire data set at coarse resolution using volume rendering, and the other for rendering detailed particle views. In terms of structural biology this size data represents about the number of protein atoms in a typical human cell. The ability to build an atomistic model of a complete human cell with all of its internal structures and interactions is, as yet, an unrealized and challenging goal for the structural biology community. However these efforts from the atom upwards coupled with those from the cell downwards, with projects like the Allen Cell Explorer⁹¹ make this goal appear feasible in the not too distant future.

Figure 10.

CellPACK model of a complete mycoplasma cell interactively visualized in CellVIEW. Two clipping cubes reveal the interior (left) and the DNA alone (right)

Computer gaming has become a multi-billion dollar entertainment industry with a strong technical focus on visualization, interactivity and engagement. Over the last several years a number of “serious” games have been developed and deployed in the educational and citizen science communities. For example, Immune attack⁹² has been developed to teach fundamentals of immunology. It is a so called “first-person shooter game that requires players to activate specific proteins to cause targeted behaviors of various white blood cells. Fold-IT⁹³ is probably the best-known biostructure- based game. It is more of an interactive puzzle that challenges users to fold up a protein structure by manipulating the unstructured polypeptide chain. It has introduced thousands of non-scientists to the intricacies of protein folding Game Engines, such as Unity3D and Torque3D, are component-based high-level software platforms designed to facilitate the development of a wide variety of sophisticated interactive computer games and experiences. Typically such development environments include resources for graphics, sound, physics, and user interface device controls. The physics components tend to include procedures for collision detection and constraint specification, such as springs between objects. These game engines can serve as effective development environments for interactive visualization and manipulation of biomolecules in modeling tools and have been used to produce programs such as UDock⁹⁴ and UnityMol⁹⁵.

The Unity3D game engine was utilized in the creation of CellVIEW⁸⁹ an interactive environment for visualizing CellPACK models. For interactive performance on large mesoscale models, CellVIEW uses screen-space GPU rendering of sphere imposters at different levels of detail (LOD) depending on distance to the camera. The rendering is applied only to objects visible in the camera’s field of view to keep the computational burden to a minimum. A scene with thousands of individual protein molecules can be difficult to interpret and navigate, so CellVIEW has a number of features to simplify and focus on specific characteristics and content. Visibility control of specified compartments or molecular ingredient types are provided. Clipping is also a useful tool to select only a portion of a larger scene or object type. CellVIEW has wide variety of clipping geometries available that can be applied to the entire scene or specific compartments or molecular ingredients to aid in selected viewing⁹⁶. Because of the multi-scale nature of these models the coloring of structural elements is also based upon level of detail⁹⁷. Thus when the camera is at a distance, the coloring of the scene is based upon the compartments or organelles in the view and, as the camera moves successively closer, the coloring can change to distinguish molecule types, then chains and finally atoms. The fact that CellPACK itself utilizes a volume-element (voxel) data structure has recently led to a new implementation that uses a novel set of GPU algorithms which unite modeling and visualization in the context of a game engine for rapid construction of mesoscale models of complex biological structures⁸⁷. These structures consist of multiple membrane-enclosed compartments including both soluble molecules and fibrous structures. Lipid membrane compartments are defined using volume voxelization of triangulated meshes, and are populated by patches of lipid bilayer pre-calculated from an atomistic molecular dynamics simulation. Soluble molecules are placed within compartments by distributing them randomly using a Halton sequence⁹⁸ on a space filling curve, which guarantees object centers will be no closer than a specified distance from one another. Fibrous structures, such as DNA, RNA or actin filaments, are created by procedure-based self-avoiding random walks. Procedural models, such as a bacterial nucleome⁸⁸, are typically treated as single ingredients and placed as such. Any resulting overlaps of molecules are resolved by a grid-localized forced-based system. The fact that the placement and relaxation of molecular ingredients is done in a highly parallel fashion on the GPU enables the interactive creation of mesoscale models of cellular environments and the production of large ensembles of such models, removing a significant barrier to structure based hypothesis generation and simulation.

CellVIEW was designed as an interactive viewer of CellPACK models, and not as a creation tool for generating new mesoscale models. In a complementary effort, using the Unity game engine with a computer painting metaphor, we have developed CellPAINT⁹⁹ to enable the production of two dimensional Goodsell style “mesoscape illustrations.” With cross-section views, three depth layers, a pallet of 2D sprites of molecular images, and circle colliders for steric exclusion, a user can draw membrane compartments, spray or brush molecules into them, and animate the scene with Brownian motion diffusion (Figure 11). Currently we are developing CellPAINT 3D in Unity, moving into a volumetric context and which could serve as a CellPACK building interface.

Figure 11.

CellPAINTing of HIV entry into a T-Cell

Perception and the User Interface

Over the last decade or so, with the continuing march of Moore’s Law and the incorporation of cameras, sensors and displays into mobile computing technology, how we interact with our computational environment has changed dramatically. Moreover, with the advancing developments of virtual and augmented reality, 3D computer printing, and the Internet of things (loT), the lines between the real and the virtual world have become blurred and our expectations of how we interact with information have expanded. While much of the fundamental technologies underlying these advances have been around for two or even three decades, they have only recently reached the level of wide-spread availability and acceptance. What, if anything, is the impact of these technologies on visualization and interaction with biomolecular structure?

Tangible and Haptic interfaces

The advent of computer autofabrication, now referred to as 3D printing, has provided an automated way to produce physical objects from computer data, and has opened up the possibility of having highly accurate material visualizations of complex molecular molecules¹⁰⁰ (see, for example, Figure 12). Physical models can convey spatial relationships and mechanisms in ways that images alone cannot. They engage perceptual and cognitive processes that go beyond the visual and bring a sense of reality and natural interaction into the process of exploration and understanding. Physical molecular models can also serve as “analogue computers” where spatial relationships between components in complex molecular interactions can be explored and manipulated. In addition, individual molecular components can be printed, and then used to build novel or flexible larger structures. For instance, this approach has been applied to create foldable polypeptide chain models^101;102 and self-assembling virus models^103;104 To broaden the use and utility of physical models in structural biology, the NIH established a repository for 3D printable models¹⁰⁵. Of course physical molecular models are constrained in ways that computer graphics visualizations are not. Every 3D printing technology uses distinct materials, each with its own mechanical properties and limitations. Therefore the range of molecular representations or color choices greatly depends on the nature of the 3D print technology used. Moreover, a physical model constitutes a fixed representation of a molecular entity and lacks the unlimited representational flexibility and manipulability of a computer visualization. Thus physical molecular models, while valuable adjuncts for examination and exploration, play a relatively limited role in molecular visualization and tend to be used mostly for communication, either in a teaching or research collaboration setting.

Figure 12.

3D printed model of influenza virus.

Force feedback devices and other haptic modalities also engage our senses of touch and proprioception. Today we find haptic displays in the form of vibrations or small taps from smart phones and watches to prompt awareness, or in video game controllers to enhance certain actions or physical effects. Touch screens, accelerometer motion sensors and depth cameras can capture finger, hand or arm gestures and body movement as input for a wide variety of computer and visualization activities. Touch screens are now commonly used to control molecular rotations, translations and scaling on laptops and mobile devices. Haptic output rendering of molecular processes, however, is more difficult. Over the years, elaborate force feedback devices have been explored for creating the feel of molecular interactions²². Such devices work both as 6 degree of freedom (DOF) input devices for positioning molecules and as 3 or 6 DOF output devices for rendering interaction forces. These devices are typically mechanical armatures with encoders and actuators in a nested succession of joints. By manipulating an attached wand moving a virtual molecule, it is translated and oriented while forces (and in some devices torques) computed from the molecule’s interactions are fed back to the user. Typically both long-range electrostatic and short-range van der Walls forces are rendered^40;106 A different use of such force-feedback devices has been to remotely control and feel the forces at the tip of an atomic scanning probe microscope¹⁰⁷. Interesting exploratory work has been done using force feedback displays in educational settings^108;109 However, even with the cost of some haptic devices now below $1000, force-feedback has not been widely adopted for molecular modeling and visualization. Their cost is still high compared to other input and output devices; the workspace of force-feedback arms is typically highly constrained; and the dynamic range of the feedback forces is limited, making it difficult to discriminate force magnitude. Hard surface rendering is especially difficult, requiring high frequency output, which can lead to uncontrolled mechanical jittering. Additionally, as a physical device that can create significant force, these arms can potentially injure a user. Recently, however, a new interest in haptics has arisen in gaming and virtual reality applications. Haptic gloves with hand and finger tracking and feedback are now becoming commodity devices¹¹⁰ but have yet to be applied in the context of molecular rendering. It still remains to be seen whether haptics and force feedback devices will be useful, useable, and adaptable to the needs of biomolecular visualization.

Virtual and Augmented Reality

Virtual Reality (VR) is typically defined as an artificial environment experienced through sensory stimuli generated by a computer. While the initial concepts and work on VR date back to the 1980s ^111;112 it has only recently become a consumer technology and new applications are appearing daily. Commercially available virtual reality headsets range in performance and price. At the low end, hand-held Google Cardboard and head-mounted smartphone display devices provide viewpoint tracking and a 360° immersive experience of pre-recorded content. At the other end are purpose-build head mounted displays (HMDs) along with motion and hand controller tracking that communicate with interactive virtual reality software running on high end GPU enabled computers. Some of the earliest explorations of VR were focused on biomolecular visualization. In one telling early experience in the 1990’s at UNC Chapel Hill, a protein crystallographer was fitted with an HMD in a large space arrayed with tracking devices to enable walking around and into a large virtual version of a protein of interest. As Fred Brooks, the project PI, has described it, the first thing the scientist requested was a chair to sit down in. Clearly, walking around inside the protein was not how she wanted to interact with her molecule and, in fact found doing so disorienting.

An experience with VR in 1998 was somewhat different. Visiting Tom DeFanti’s Electronic Visualization Lab (EVL) at U. Illinois at Chicago. I had imported our early mesoscale model of a 75nm cube of blood plasma (Figure 13), into the CAVE¹¹³ a cubic space with stereoscopic projections on surrounding walls and ceiling. Exploring our mesoscape without the need of a bulky tethered HMD, made walking among the boulder sized proteins and sticking my head inside the poliovirus capsid an engaging and inspiring experience. However, as I couldn’t afford the space for, or the cost of, a CAVE for our routine visualization environment, I didn’t pursue its application to structural molecular biology at the time.

Figure 13.

Mesoscale blood plasma model, that was imported into VR in the CAVE in 1998.

Even today, VR is groping around in the molecular world, to find a “killer app” for structural biologists. Would implementing a widely used molecular visualization environment like PyMOL or Chimera in a virtual reality environment entice a lot of users to drop their traditional molecular displays? On the plus side, the VR environment can provide a strong sense of presence with the molecular model. Employing hand tracking to position and pose a molecule in VR eases cognitive load, and can enable more accurate placement while facilitating “what-if” type exploration. Perceptually, the combination of spatial cues from head motion coupled with stereoscopy along with the person centric immersive perspective increase a sense of realism and true three dimensionality. On the negative side, the head sets themselves, even though they continue to improve, are still intrusive to wear, and are physically restrictive, with cables tethered to the computer. Also, for an immersive experience that doesn’t leave one dizzy or motion sick, fast image update rates (over 90fps), high resolution, and low latency motion response, as well as correctly computed stereoscopy, are critical. Most importantly, the sense of removal from the real world when immersed in the virtual, can be disorienting, and feel disembodied and isolating after extended use.

Since game engines are designed to enable running software on multiple platforms, we decided to implement our prototype CellPAINT 3D program for VR headsets (Figure 14). We have found that the ability to place objects in an immersive volumetric space provides a unique user experience. CellPAINT VR is still in early development, and the painting metaphor is evolving into a 3D molecular mesoscale model-building environment. Many issues, such as the interactive building of molecular complexes and effective “way finding” and navigation in complex environments still need to be improved.

Figure 14.

Virtual Reality implementation of CellPAINT 3D. Video screen shows part of immersive HMD display.

Augmented Reality (AR) blends the real and cyber world together using a number of different types of interfaces¹¹⁴. Mobile devices are now capable of running augmented reality applications, using their built in camera to recognize some aspects of the surrounding world, and then inserting computer generated text or 3D graphics into the scene. The computer augmentation is viewed on the hand-held display in the context of the real-world camera view. Depth information acquired by either camera motion or optics convincingly places the generated graphics into the scene. Pokemon Go, a popular game, introduced a wide audience to this technology. More capable AR systems have been developed with head-mounted displays that use computer projections onto a transparent visor in front of the eyes, with the real world still in view. The Microsoft Hololens is an example of this class of AR display device. The key to augmented reality is the recognition and tracking of real-world objects or environments, and the ability to convincingly display computer generated material within that context. The strong advantage of AR over VR is that the user operates in the real world but can perceive and interact with computer generated visualizations.

Several years ago, our interest in the use of 3D printed molecular models, led us into the area of Augmented Reality. Through the use of an early open source augmented reality toolkit (ARToolkit¹¹⁵), with coded markers on objects for camera recognition, we developed Tangible Molecular Interfaces^116;117. By attaching these markers to molecular models, we could track their motions with a camera, and augment the molecular models with additional representations and computed properties, such as surrounding electrostatic fields. Since multiple molecular models could be tracked in the computer we could continuously monitor distances between them and interactively compute and display energetics of molecular interaction (Figure.15). To this point there has been no widespread adoption of Augmented Reality in the field of structural molecular biology. As costs continue to drop, and as form and function of the devices become less intrusive and easier to use, these technologies could be especially effective in scientific collaboration, where multiple people can view and interact with the same virtual molecular entities while still being engaged in the real physical environment.

Figure 15.

Augmented reality display of 3D printed flexible polypeptide folding model. Tracking marker is seen near the top. Computer augmented amino acids can be manipulated and distances between them measured.

Prospects/ Summary

Scientific and technical trends appear to be clear. Structural molecular biology continues to expand into more complex systems and integrative approaches, and the path from the atom to the living cell appears ever more approachable. Information visualization of non-structural data continues to evolve and combine with structural visualizations. Computer power and the utilization of that power with more effective algorithms and more capable physical devices continue unabated. Digital communication and the extent and pervasiveness of networked people and machines bring together new communities and collaborations.

Artificial Intelligence (Al) and machine learning are perhaps the most widely anticipated technologies for the near future, and are being used in ever-broader applications, including molecular biology. Since Al is useful for pattern discovery and image interpretation, we are beginning to see the coupling of Al algorithms with interactive recognition of various imaging technologies, such as electron tomographic annotation¹¹⁸. Pattern recognition can also help refine parameters for creating mesoscale molecular models of cellular features, such as membrane and fiber shapes and distributions. With the structural biology literature expanding at exponential rates, machine learning used in text and image processing could assist greatly in developing and curating consensus views of molecular componentry and distributions in cellular systems. However, since the function of visualization is to put human in the loop, will artificial intelligence make visualization obsolete? It is my belief that it will be a long time before human inspiration, insight and fresh concepts derived from visualization and interaction are no longer central to the progress of structural molecular biology.

Highlights.

This perspective traces the evolution of structural molecular biology visualization from physical models of the earliest protein structures to immersive interaction of mesoscale models of cellular environments.