The OptIPuter microscopy demonstrator: enabling science through a transatlantic lightpath

Abstract

The OptIPuter microscopy demonstrator project has been designed to enable concurrent and remote usage of world-class electron microscopes located in Oxford and San Diego. The project has constructed a network consisting of microscopes and computational and data resources that are all connected by a dedicated network infrastructure using the UK Lightpath and US Starlight systems. Key science drivers include examples from both materials and biological science. The resulting system is now a permanent link between the Oxford and San Diego microscopy centres. This will form the basis of further projects between the sites and expansion of the types of systems that can be remotely controlled, including optical, as well as electron, microscopy. Other improvements will include the updating of the Microsoft cluster software to the high performance computing (HPC) server 2008, which includes the HPC basic profile implementation that will enable the development of interoperable clients.

1. Introduction

Electron microscopes are expensive and often unique facilities that cannot easily be replicated across the globe, but which increasingly have the capability for remote access. A world-class biomedical research group at the University of California San Diego (UCSD) and a similarly world-leading materials science group at the University of Oxford have independently developed new imaging instruments that should revolutionize the way we obtain three-dimensional information at the highest possible resolution. The transatlantic collaboration described here developed new ways to share views of the complexities of the nervous system and of complex nanomaterials. The science drivers include the identification of materials to enhance the behaviour of automotive and industrial catalysts and the creation of a dynamic brain atlas using the collected images with correct referencing within the whole system.

Both groups of researchers recognized the overlap of their developments and the mutual benefit of working together to refine their new imaging instruments and methods of building and displaying the three-dimensional datasets. At Oxford and the UCSD, computer scientists, who have a long history of working with these imaging researchers, recognize this as an ideal opportunity to deploy and perfect high-bandwidth-based collaboration infrastructure replete with cluster computing, distributed databases and cluster-based multi-screen image exploration environments (figure 1). Specifically, the project created an infrastructure for collaboration whereby instruments, databases, ultra-high-resolution visualization systems and computational grid resources are integrated via high-bandwidth, low-latency networks and leading edge information technology/middleware technologies.

Figure 1.

The microscopy distributed laboratory demonstrator.

2. Background

Both biological and materials science electron microscopy are challenged by a need to increase the collaborative use of rare and specialized instruments. Modern electron microscopes, in particular those participating in this project, are increasingly unique specialized instruments for which hosting institutions have provided large amounts of upfront capital investment. It is therefore extremely important that they are used as often as possible. As the demand for the use of these instruments and the expertise surrounding them continues to rise, there is equal need for solutions to more effectively connect researchers to these unique resources, to one another and to the data. This is especially true across continental boundaries where travel costs are becoming increasingly prohibitive. However, for remote collaborations to be truly successful, the interactivity of remote control and the ability for geographically distributed researchers to richly interact as if they were collocated is paramount.

For electron microscopy, this goal is complicated by a number of constraints. The imaging techniques are extremely data intensive, and the ability to process the three- and four-dimensional data at the time scales suitable for interactive collaboration typically requires more computational horsepower than is available at most advanced microscopy facilities. The further need for high-definition video to be able to ‘steer’ the instruments in real time requires next generation networking performance that spans oceans, and more importantly, navigates last mile hurdles. The exploration and analysis of massive data, especially to assist real-time control, requires visualization technologies that can integrate and interactively display ultra-high-resolution data from multiple sources and of multiple modalities (three- and four-dimensional streaming, etc.).

The microscopy distributed laboratory demonstrator developed here provides a total collaboration solution for electron microscopy. The project harnessed the expertise of leading researchers in tele-instrumentation, cyberinfrastructure development, optical/lambda networking, advanced visualization and microscopy. It also demonstrated how a Microsoft- (MS-) enabled infrastructure can provide a solution for collaborative tele-instrumentation that scales beyond the singular capabilities that will be leveraged for this effort.

Specifically, the demonstrator was built using two of the world's most advanced electron microscopes in the US and UK. Hence, the shared use of these resources represents a unique opportunity to link biological and materials science technical expertise across two leading resources. It has also provided a forum by which materials scientists and biologists can share expertise and experience across disciplines.

Leading computer scientists, biomedical scientists and materials scientists spearheaded the effort, with a multidisciplinary collaboration including the UCSD, specifically it is the National Center for Microscopy and Imaging Research (NCMIR), the California Institute for Telecommunications and Information Technology, the Materials Science department of the University of Oxford and the Oxford e-Research Centre (OeRC).

The OeRC is in central Oxford based in the e-Science building; however, the microscope is based at the Begbroke Science Park, which is 8 km from the centre of Oxford, providing a clear illustration of the issues that can arise within the ‘last mile’. Hence, it was important to create a seamless network within the institution providing the infrastructure to enable collaborations with scientists, both at the Begbroke site, as well as at the OeRC.

3. The microscopy distributed laboratory architecture

The architecture of the cyberinfrastructure for the demonstrator is shown in figure 2.

Figure 2.

Demonstrator architecture.

The challenges in developing a suitable architecture included the fact that experiments are performed in real time and, as such, need high-resolution inter-site video communication of a standard that can be reached only using high-definition television (HDTV) standards. Also, the processing of raw output images must be done sufficiently quickly that the experiment could be rerun if necessary, and the processed results need to be disseminated to all the collaborators as soon as available. The raw data should also be archived and annotated appropriately to enable reuse later. To establish data integrity, any further processing and annotation must also be done only on true copies, so that the raw information is never at risk of loss. In this section, we present the components of the architecture driven by these requirements.

(a) Electron microscopes

The microscopy facilities that are involved in the project are some of the most advanced in the world. The instruments at both San Diego and Oxford represent the state of the art in intermediate voltage and aberration corrected geometries for transmission electron microscopy, with the instrument in Oxford being the world's only double corrected instrument (for both spherical and chromatic aberrations). At NCMIR, the facility includes the largest charge coupled device camera ever constructed, with specialized sample mobility for 360° degree optical sectioning. There are several factors that make remote operation of these instruments desirable, including the remote physical location of the instruments due to a desire for ultra-stable (in terms of vibration, temperature and electromagnetic stability) operating conditions.

In this case, the challenges were to enable fully duplexed real-time instrument control (optics, sample positioning, ancillary devices), to provide a high-bandwidth persistent connection for multiple image feeds (more than 1 MB s⁻¹), with appropriate image compression of relatively high noise level data, to provide a multi-level user interface that could operate in across heterogeneous operating systems, provision for multiple remote users with appropriate authentication and security. These requirements are dealt with below.

(b) Lambda networks and the OptIPuter project

The magnitude of data resolution and the requirement to view data in full resolution, often juxtaposed in context with other data, required the use of advanced, next generation environments. The lambda network that was assembled was critical to preserve the resolution of data sizes and the HDTV streams (from multiple collaborative sites). An uncompressed 1080p HDTV stream at 30 frames per second requires approximately 1.5 gigabits per second (Gbit s⁻¹) of bandwidth. For real-time collaborative experiments with multiple camera and instrument feeds, the bandwidth requirements can be several Gbit s⁻¹. The project extended the switched LambdaGrid optical network project, OptIPuter (Smarr 2007), between UCSD and the University of Illinois Chicago, to the University of Oxford. The OptIPuter was a National Science Foundation project that was aimed at developing an advanced distributed computing infrastructure for collaborative data exploration in the fields of biological and Earth sciences. The project aimed at experimenting with the intriguing idea of user controlled light paths where users, for the first time, get to schedule dedicated network resources and fibre between sites for experiments. This potentially allows a group of researchers to custom tailor a scientific visualization pipeline with dedicated bandwidth of several Gbit s⁻¹ for moving large data. This approach would allow exploration of large data collaboratively by research groups separated by large geographical distances. Once the experiment is over, the resources would be freed and reconfigured for a different purpose.

A lambda network allows the use of multiple optical wavelengths to provide a point-to-point high-bandwidth and low-latency communication channel along fibre optic cable (S. Wallace, Lambda Networking, private publication, http://www.anml.iu.edu/PDF/Lambda_Networking.pdf). This end-to-end connection is referred to as the lightpath. Within the UK, there is an experimental optical network, UK Lightpath (JANET UK Lightpath, http://www.ja.net/services/lightpath/index.html), which comprises a 10 Gbit s⁻¹ backbone to selected points in the UK and connects to global optical networks via 10 Gbit s⁻¹ links to Chicago (via StarLight (US StarLight, http://www.startap.net/starlight/ABOUT/)) and Amsterdam (NetherLight (NetherLight Network, http://www.surfnet.nl/nl/Thema/netherlight/Pages/Default.aspx)). The project leveraged the global lambda integrated facility (http://www.glif.is/) to enable the transatlantic network connection.

(c) The computational requirements

The computational requirements for the project are many fold and include the necessity for processing the output images in real time. With the technological advances, the systems are also able to look at many more data-intensive modes of operation, e.g. tomography, exit wave reconstruction. It is therefore necessary to connect both high performance computing (HPC) systems to do the real-time image enhancement and processing within high-resolution visualization systems. To this end, the team developed a Web service to expose the OeRC MS Compute Cluster Server (CCS; http://www.microsoft.com/HPC/default.aspx) to handle job submission (via MS message passing interface) and execution of the parallel processing applications.

(d) Tile walls, scalable adaptable graphics environment (SAGE) and videoconferencing

Tile displays were in turn necessary to provide the pixel ‘real estate’ to visualize the data in a manner that preserves both detail and wide-field context. The typical configuration was 1920×2400 pixels wide-screen displays arranged in a 3×4 array with an active viewing area of 4800×4096 pixels, all controlled by Dell Precision 309 systems with NVIDIA graphics cards and running Windows XP.

The scalable adaptable graphics environment (SAGE; Jeong et al. 2005) a graphics streaming architecture for supporting collaborative visualization and high-definition displays was ported to run on the Windows XP platforms. SAGE is designed to allow collaborators to simultaneously run multiple applications and to share them by streaming the pixels of each application over ultra-high-speed networks to large tiled displays.

The original plan in the project was to integrate the SAGE libraries with Windows tools for image display including Windows Media Player and high-definition conferencing using Conference XP. Each participating site has been installed and configured with high-resolution videoconferencing equipment, so that the microscope control rooms and analysis suites could be in constant communication. One issue that arose with SAGE was that all of the software that runs on the display wall must have additional libraries built into them to enable splitting the graphical output to display across the nodes of the tile system. Owing to problems with incorporating these libraries in the MS products, an open source product, VideoLAN Client (VLC; VLC Media Player, http://www.videolan.org), was used instead. This also allowed significant flexibility in the material that could be shown, including the raw microscope output images, the high-resolution video streams from the control and analysis suites, as well as static processed images.

(e) The middleware

The middleware development was a software project with a number of elements, including the installation of authentication and appropriate software systems in Oxford and the UCSD to allow remote access to resources. Figure 3 indicates the software architecture of the system. As described above, SAGE provided an important component of the middleware—and equally important was an existing software system developed by NCMIR, the generalized telemicroscopy system (GTS; Molina et al. 2005). The GTS was installed at Oxford to enable preliminary remote access to the microscope and to integrate control of the instrument detectors and accessories. It extends the instrument control systems from classic client–server applications to include the full functionality and interoperability of web (and grid) services. It provides the authentication required to provide secure access to the microscopes and data. Figure 4 provides the detailed architecture of GTS within the context of this project.

Figure 3.

Software architecture.

Figure 4.

GTS architecture.

(f) The interface

An additional collaborative component of the project was the development of a Microsoft foundation technology-(MSFT-) based ‘Portal’. Developed as a ‘desktop application’ using MSFT C#, this portal provides users with remote monitoring capabilities and the ability to securely access instruments and computational resources. Using GAMA Web services, this Portal securely logs the user into an environment to: (i) launch the MS-based telemicroscopy control (for both the UCSD and Oxford instruments), (ii) perform image processing on WinCCS clusters (via NET services), and finally (iii) remotely preview (via LamdaCAM) and control (via MagicCarpet) remote TileDisplays. This interface is unique as it employs the usage of the GAMA user account management system, which is a native Java service. Its integration with our MSFT C# desktop application further demonstrates the heterogeneous nature of our software infrastructure.

4. Demonstrating at SC07

One aim of the project was to run a demonstration at the SC07 conference, which, for the purpose of the demonstration, was also connected into the dark fibre network at a bandwidth of 10 Gbit s⁻¹. All the compute resources and instruments were part of the same network subnet at Reno, the UCSD and both Oxford sites. During the conference, the team used the system to demonstrate a researcher operating the microscope at NCMIR, working from the tile display situated in the OeRC, while simultaneously communicating with a scientist situated in Reno describing the outputs to the assembled visitors.

Submission of tasks for image enhancement was shown, as well as the output from one of these jobs showing the application of the MS CCS system to do complex scientific tasks. All the while, the groups at Oxford, the UCSD and Reno were able to communicate across the world confirming operations, etc., performing remote operation of multi-million pound equipment that would otherwise be impossible. The demonstrations ran for the whole week of the SC07 conference and included controlling the microscope from Reno and demonstrations of high-resolution results displayed on the tile wall located on the show floor.

Following on from the successful demonstration, the system has been used a number of times, primarily to view a remote instrument in isolation. This still allows the sharing of unique microscopes between Oxford and San Diego, but minimal use of the concurrent capability has been made. Further developments are needed on the stability of the developed software to harden it into a production service. This has some effects on the possible increase in the usage of the facilities, since they still require an operator to be present, with time zone differences, which is more difficult to achieve. The system could be quite realistically used between disparate components of a large multi-site university with ease, giving increased efficiency of the services.

5. Conclusions

The benefits of internationally connected lambdas, MS OptIPuter-enabled visualization systems, and MS Windows applications and clusters were demonstrated in the context of a fully integrated infrastructure for the shared use of unique imaging instruments coupled to processing workflows for the three-dimensional characterization of connections in the brain and advanced materials.

This approach of software enabling access to complex resources on different continents has many different applications outside microscopy. It has also illustrated the application of MS technologies through an interface with which a researcher will already be familiar, which can only speed the research process.

The resulting system is now a permanent link between the Oxford and San Diego microscopy centres and will form the basis of further projects between the sites and expansion of the type of systems that can be remotely controlled, including optical, as well as electron, microscopy. Most importantly, it has enabled the scientists involved to plan and execute science experiments that they otherwise would not be able to achieve.