The Application of JPEG2000 in Virtual Microscopy

Abstract

Virtual microscopy (i.e., the viewing of entire microscope specimens on a computer display) is becoming widely applied in microscopy teaching and clinical laboratory medicine. Despite rapidly increasing use, virtual microscopy currently lacks of a universally accepted image format. A promising candidate is JPEG2000, which has potential advantages for handling gigabyte-sized virtual slides. To date, no JPEG2000-based software has been specifically suited for virtual microscopy. To study the utility of JPEG2000 in virtual microscopy, we first optimized JPEG2000 code-stream parameters for virtual slide viewing (i.e., fast navigation, zooming, and use of an overview window). Compression using ratios 25:1–30:1 with the irreversible wavelet filter were found to provide the best compromise between file size and image quality. Optimal code-stream parameters also consisted of 10 wavelet decomposition levels, progression order Resolution-Position-Component-Layer (RPCL), a precinct size of 128 × 128, and code-block size of 64 × 64. Tiling and the use of multiple quality layers were deemed unnecessary. A compression application (JVScomp) was developed for creating optimally parameterized JPEG2000 virtual slides. A viewing application (JVSview) was developed specifically for virtual microscopy, offering all of the basic viewing functions. JVSview also supports viewing of focus stacks, embedding of textual descriptions, and defining regions of interest as metadata. Combined with our server application (JVSserv), virtual slides can be viewed over networks by employing the JPEG2000 Interactive Protocol (JPIP). The software can be tested using virtual slide examples located on our public JPIP server (http://jvsmicroscope.uta.fi/). The software package is freely downloadable and usable for noncommercial purposes.

BACKGROUND

Virtual slides are digitized versions of whole microscope specimens that can be viewed on a computer display.¹ Virtual slide image files can be accessed from the computer’s hard disk, transportable media (e.g., a USB memory stick or a DVD), or, more practically, over networks.² Within the internal network of a hospital or pathology department, virtual slides can be used for case meetings, slide seminars, and didactic live audience presentations.³ By allowing access over the internet, virtual slides can be used more widely for national and international conferences and in interlaboratory quality assurance programmes.⁴ Feedback from microscopists shows that virtual microscopy can be regarded as a significant improvement over digital snapshot images and the use of multi-headed microscopes.^5,6

To create virtual slides that resemble real microscopy viewing experiences, specimens must be scanned at high optical resolution. Image resolutions from 0.20 to 0.40 μm per pixel are considered necessary for sufficient image quality.⁷ Since microscope specimens are often up to 20 × 30 mm in size, a virtual slide can contain up to 40 gigabytes of uncompressed image data (with three 8-bit colour channels). When virtual slides are applied in cytopathology or hematology, a higher optical magnification (scanning with oil-40×, oil-60×, or oil-100× lens) is often needed.⁸ This multiplies the amount of image data produced when compared to standard scanning. Moreover, when several focus planes (along Z-axis) are digitized, the resulting data amount is further multiplied by the number of planes. Due to the large size of the virtual slide files, it is not possible to use conventional image viewing software (such as Photoshop®), which requires the image data to be loaded entirely into computer’s memory (RAM). For this reason, the virtual slide viewing systems described to date apply the on-demand principle, that is, the viewing software decodes and displays only a user-requested area of the virtual slide at the requested resolution. When using JPEG-compressed images, this requires splitting the virtual slide image data into tens of thousands of very small image files, which are uploaded to the client on demand (e.g., Zoomify).⁹ Proprietary image formats provided with commercial slide scanners (e.g., Mirax by Carl Zeiss MicroImaging, USA; dotSlide by Olympus UK, UK; and ScanScope® by Aperio Technologies, USA) also follow this principle in their web viewing solutions. Due to competition, it is unlikely that any of the proprietary industry-based image formats will become universally accepted in virtual microscopy.

A promising candidate for a universal image format for virtual microscopy is JPEG2000, which is a family of standards, developed by the Joint Photographic Experts Group.¹⁰ It is based on a wavelet-based compression scheme that is defined in the core part of the standard series (published as International Standard ISO/IEC 15444-1 | ITU-T Rec. T.800).¹¹ JPEG2000 provides many features that support scalable and interactive access to large-sized images such as virtual slides. These include an efficient and unified compression architecture, especially at low bit-rates, resolution and quality scalability, region of interest coding, spatial random access, and effective error resiliency.^12–14 Currently, the JPEG2000 standard consists of 11 parts, which describe techniques that are useful in various areas of imaging. Part 1 (Core Coding System),¹¹ Part 2 (Extensions),¹⁵ and Part 9 (Interactivity Tools, APIs and Protocols)¹⁶ are the most essential for virtual microscopy. The first part specifies the JPEG2000 code-stream syntax and the JP2 file format (“jp2” as the common file extension), which offers, for example, the possibility of adding textual metadata into the image header. The second part specifies the JPX file format (“jpx” and “jpf” as the common file extensions), which contains more advanced features, such as building stacks of images (e.g., for specimens scanned at multiple focal depths), and associating textual and graphical metadata with spatial regions. The ninth part defines the JPEG2000 Interactive Protocol (JPIP), which provides a client–server architecture for network-based remote viewing of JPEG2000 family images.¹⁷ Currently, web browsers do not have direct, built-in support for JPIP. For this reason, internet-based viewing of virtual slides over JPIP requires external viewing software, which is, for example, invoked automatically from the browser upon clicking on a virtual slide URL link.

Although JPEG2000 has theoretical advantages for application in virtual microscopy, no efficient application software has been described so far. The computational demands of virtual microscopy images differ to a large extent from that of radiology and other medical imaging because of the larger size of the image files (up to tens of gigabytes). In this study, we first searched for an optimal parameterization of JPEG2000 for virtual microscopy and constructed a JPEG2000-based software package entitled JVS (for JPEG2000 Virtual Slide), consisting of image compression, viewing, and network server applications. The performance of JPEG2000 was evaluated using JPIP image serving speed and compression execution time as outcome measures. In addition, the performance of our network server application (JVSserv) was compared against the commonly used virtual slide server solution Zoomify.⁹ Finally, we present a model for a simple, scalable, and inexpensive JPEG2000-based virtual microscopy system.

MATERIALS AND METHODS

Image Acquisition

We acquired test images by using a Zeiss Axioskop40 microscope (Carl Zeiss MicroImaging, USA) which was equipped with 10×, 20× and oil-40× objectives and a motorized specimen stage (Märzhäuser Wetzlar GmbH, Germany). The images were captured with a CCD colour camera (QICAM Fast, QImaging, Canada; 24-bit colour depth; resolution 1,388 × 1,036 pixels, pixel size 4.65 μm). The camera was attached to the microscope with a 1× phototube. Image acquisition was controlled by the Surveyor imaging system (Objective Imaging, UK, software version 5.03) running on a standard Windows® XP workstation. The Surveyor software controls for stage and focus movements matched with automated image acquisition. In addition to standard histological samples, cytological slides were scanned using a multiple focus layer image acquisition option (Z-stack). Image tiles (up to 5,000 per scanned slide) were primarily saved in an uncompressed file format (BMP) and stitched as a contiguous montage either using the built-in function of Surveyor, or using the LargeMontage plugin¹⁸ of the publicly available image analysis software ImageJ.¹⁹

JPEG2000 Code-Stream Parameter Optimization

The JPEG2000 code-stream parameter optimization was based on theoretical assumptions of the JPEG2000 code-stream properties, visual assessment of the time needed to initially load the image, the time needed for image refreshment during magnification change and navigation within the specimen, and to lesser extent, the compression execution time. Tests were performed using the highly configurable Kakadu kdu_compress and kdu_show applications, version 5.2.2.²⁰

JPEG2000 Performance Evaluation

The performance of JPEG2000 was evaluated using JPIP image serving speed and compression execution time as outcome measures. The tests were carried out on a Windows® XP workstation, equipped with a dual-core processor, 3 gigabytes of RAM, two ATA hard disks, and a 100 Mbit network link. The used main test virtual slide image can be seen on our website (http://jvsmicroscope.uta.fi/examples/). The image had 42,865 × 57,222 pixels, three 8-bit colour channels, totalling 6.9 gigabytes uncompressed, and it was stored as a binary encoded PPM file. An additional set of test images was created by extracting sub-resolutions from the original image, halving the resolution in each step.

The compression tests were performed using our JVScomp application. During these tests, we monitored the processor load, RAM usage, disk usage percentage, and the average disk read and write queue lengths. The image serving performance of JPIP was evaluated with our JVSserv application, in comparison to the Zoomifyer EZ™, version 3.0.⁹ While JVSserv is a standalone server application, Zoomifyer requires an external HTTP server underneath it. The Apache HTTP server, version 2.2.4,²¹ was selected for this. The evaluation and comparison were performed by simulation, in which 10 local area network workstations were each simulating 10 clients, yielding a 100-client pool. Each client had their own test image, which was duplicated from the main test image. Server and client caches were disabled. During these simulations, we monitored the server’s average disk read queue length, processor load, network bandwidth usage, RAM usage, and subjectively evaluated the clients’ viewing interactivity.

Software Package Development

The demonstrational software package was written with the C++ programming language and built for 32-bit Windows® platforms, but can be run under a 64-bit Windows® platform as well. The package consists of three applications: JVScomp—a compression application for creating optimally parameterized JPEG2000 virtual slides; JVSview—a viewer application capable of viewing both local and remote JPEG2000 virtual slides; and JVSserv—a JPIP server for remote serving of JPEG2000 virtual slides. JVScomp is built on the ECW JPEG 2000 SDK,²² while JVSview and JVSserv are built on the Kakadu JPEG2000 Framework.²⁰

Results

JPEG2000 Code-Stream Parameter Optimization

To fully utilize the advantages of JPEG2000 in virtual microscopy, we first searched for an optimal combination of JPEG2000 code-stream parameters. The findings are summarized in Table 1. In virtual microscopy, lossy compression is mandatory due to the large amount of image data. It is shown that highly compressed JPEG2000 images can be used for pathological diagnostic studies.²³ We found compression ratios ranging from 25:1 to 30:1 in areas containing cells or tissue sections to produce virtual slides without disturbing visible image compression artefacts at the original image resolution. A comparison of the image quality between different JPEG2000 lossy compression ratios is demonstrated on our website (http://jvsmicroscope.uta.fi/examples/).

Table 1.

JPEG2000 Code-Stream Parameters Found Optimal for Virtual Microscopy

JPEG2000 Code-Stream Parameter	Parameter Value
Compression ratio (lossy)	25:1 to 30:1
Wavelet filter	Irreversible
Wavelet decomposition levels	10
Tiling	Not needed
Tile-parts	1
Precinct size	128 × 128
Code-block size	64 × 64
Progression order	RPCL
Quality layers	1
PLT pointer marker segments	Inserted always

Of the two alternative wavelet filters (reversible and irreversible), the irreversible filter yielded somewhat better visual quality, but at the expense of compression speed. The Discrete Wavelet Transform (DWT) process decomposes the original image into several low-resolution versions. To extract an overview image, we found 10 DWT levels to be sufficient for even the largest virtual slides, although this creates redundant resolution levels for smaller slides. Tiling and the use of multiple tile-parts were not found to be useful in virtual slides. Technically, an untiled JPEG2000 image actually contains a single tile whose size is identical to the image size. We found precincts to be a more useful alternative for partitioning the JPEG2000 code-stream for random access. A precinct size of 128 × 128 was a suitable average for all resolution levels. Precincts are further divided into code-blocks, the optimal size of which was 64 × 64. The progression order defines how packets—the basic segments of a JPEG2000 code-stream—are prioritized. We found the Resolution-Position-Component-Layer (RPCL) order most useful, since it allowed the fastest access to different image resolutions, as well as fast spatial random access. No particular advantages were found for multiple quality layers, since the gradual image quality improvement after navigation or zooming became noticeable only with very slow-speed JPIP connections. The JPEG2000 code-stream headers are composed of so-called markers and marker segments, which are used, for example, to specify the image size and to offer pointers to the code-stream. To further increase the code-stream random access, we found it advantageous to use packet length pointer marker segments within the tile-part header (PLT).

JPEG2000 Compression Performance

We performed various compression speed tests to ensure that JPEG2000 can be used within a real virtual microscopy scanning system. The duration of the compression process was linearly dependent on the original virtual slide image size. RAM usage was relatively low, as during the compression of a 7-gigabyte virtual slide the memory footprint of JVScomp constantly remained around 70 megabytes. JVScomp reached a compression speed of 50 gigabytes per hour on a dual-processor workstation. The limiting factor was the processor performance, as the source hard disk could have effectively processed over twice the number of read requests.

JPIP Image Serving Performance

We evaluated the serving performance of the JPIP protocol using our JVSserv application. In addition, a comparison was made to the commonly used Zoomifyer EZ™ virtual slide server solution. On both systems, virtual slide serving was limited by the speed of the hard disk; the processor load was constantly under 50%, as was the RAM allocation. The network bandwidth (100 Mbit/s in our tests) was utilized only about 25% of the maximum transmission rate. As the number of clients increased, the average disk read queue lengths quickly reached the effective performance limits (an average of 2.0 was considered to be a suitable limit).

With the main test virtual slide, which was duplicated for every client, JVSserv could effectively handle about 30 simultaneous clients, while the Zoomifyer (run on top of the Apache HTTP server) had its limit at around 20 simultaneous clients. The difference can be explained by the fact that Zoomifyer splits the image into tens of thousands of small JPEG files, which the HTTP server (Apache) then serves to the clients. Because of caching policies, both servers would have benefited for clients that browse the same image simultaneously. Thus, assuming that some clients are usually viewing the same image, JVSserv could easily serve 50 simultaneous clients.

Furthermore, JVSserv features a load balancing function, which can be used to distribute clients to several sub-servers. Thus, we estimate that a standard JVSserv workstation behind a 100 Mbit/s network link could handle 200 to 300 simultaneous clients when supplemented with 4 to 5 sub-servers.

Software Package Description

We developed the demonstrational JVS software package to illustrate the utility of JPEG2000 in virtual microscopy. The package is designed primarily for Windows® 2000 and XP, and is Windows Vista™ compatible. JVScomp is licensed under the GNU General Public License, while the binaries (i.e., the actual executable programs) for JVSview and JVSserv are released for free, non-commercial usage. The software package can be obtained from our website (http://jvsmicroscope.uta.fi/). Our website also features a public JPIP server with several virtual slide collections, which can be viewed using JVSview.

JVSview. This software is a JPEG2000 virtual slide viewer that works identically both in local storage and JPIP-based virtual slide viewing (Table 2). JVSview displays an overview of the entire sample and a detailed microscopic view. The magnifications have been set to mimic commonly used microscope objectives. The user can navigate around the sample using either the arrow keys, mouse panning, or by clicking on a location in the macroscopic view. The application allows the user to interactively adjust brightness, contrast and colour saturation of the main window. Multiple display devices can be used in full screen viewing mode. Viewing interactivity is dependent on the processor performance of the client workstation. Acceptable speed can already be achieved with conventional single-processor computers, although JVSview has an automatic support for multi-processor environments. To illustrate new functionalities of virtual microscopy, made possible by advanced features of JPEG2000, we included a focus feature, which enables switching between alternative image layers, each representing the cells or tissue scanned at different focal depth (see the examples on our website http://jvsmicroscope.uta.fi/examples/). We also included the possibility to embed short textual data conveniently within the header of the virtual slide file. The textual data, called metadata, can contain information on the scanning resolution, organ and histopathologic diagnosis, and the staining method, for example. The metadata is written in XML format and its structure is described formally with an XML schema;²⁴ it can be added and edited within JVSview. The metadata is primarily for the purposes of image collections, where the associated patient data is irrelevant (e.g., those used in teaching). JVSview also allows specifying and storing regions of interest (ROIs), which can then be retrieved from a ROI list by the clients. JVSview also provides a functional link with ImageJ,¹⁹ which is an efficient, feature-rich, public domain image analysis application. The link to ImageJ is an easy way to measure lengths and areas as well as to count particles (e.g., cells and nuclei) using a click-and-count feature.

Table 2.

Main Features of the JVSview JPEG2000 Viewer Application

List of Features
Local image viewing & remote image viewing over JPIP
Overview window representing the entire specimen
Magnifications resembling common microscope objectives
Interactive panning with mouse or keyboard
Image brightness, contrast & colour saturation adjustments
Multi-display full screen viewing
Support for multiple image layers (e.g., focus layers)
Storing information and regions of interest as XML metadata
A functional link with image analysis software (ImageJ)

JVSserv. This software is a JPEG2000 virtual slide server application, which utilizes the JPIP protocol as its data transfer technique (Table 3). It is based on the kdu_server application by the Kakadu Software.²⁰ It has been modified by adding a simple graphical user interface, and by modifying image sample count restrictions and the messaging system. Therefore, JVSserv should only be used with the JVSview viewer application. JVSserv supports multiple simultaneous client connections and uses a standard FIFO (First In, First Out) caching policy, which is used to maintain the most frequently accessed data in memory. JVSserv also supports load balancing, which means sharing the main server’s workload to several sub-servers, thereby allowing the system to handle more simultaneous clients.

Table 3.

Main Features of the JVSserv JPEG2000 Server Application

List of Features
Remote image serving over JPIP
Graphical user interface for administration
Support for multiple simultaneous clients
Multiple server clustering (load balancing)

JVScomp. This software is a JPEG2000 virtual slide compression application capable of creating virtual slides that are optimized for viewing and serving with JVSview and JVSserv (Table 4). It follows the parameterization guidelines described earlier, and it is capable of compressing various input file formats, such as PPM, BMP, and JPEG. It supports dual-processor workstation environments, which are detected automatically. JVScomp utilizes the ECW JPEG 2000 SDK compression algorithm,²² which employs a rate control policy that yields as efficient a compression as possible. If an image contains areas that are substantially responsive to compression, for example, homogenous areas of virtual slide glass background with no tissue section, a higher compression ratio is applied to these areas. Thus, a greater overall compression ratio and a smaller file size can be achieved.

Table 4.

Main Features of the JVScomp JPEG2000 Compression Application

List of Features
Optimised parameterisation for virtual microscopy
Efficient rate control policy for virtual microscopy
Support for dual-processor environments
Support for various input file formats

Discussion

Today many manufacturers of virtual slide scanners use proprietary image formats, which preclude image exchange between different systems. Support of proprietary non-standard image formats may also be short-lived. For these reasons, the use of an open image standard is a prerequisite for large-scale virtual slide collections that are supposed to be accumulated and utilized for years or even decades. Since technical development is likely to continue rapidly in slide scanners, it is advantageous if image acquisition can be decoupled from image archival and delivery. This ensures that virtual slides generated with older generation equipment are compatible with newer ones. All these conditions are met with the JPEG2000 image format, which, based on its advanced features, is in many ways an ideal choice for virtual microscopy. The present study defined optimal code-stream parameters, which are essential for fast viewing of JPEG2000 virtual slides.

Compression of virtual slide image data into JPEG2000 format is a computationally intensive process. As shown in our experiments, the limiting factor with a conventional dual-core processor environment is the processor performance. However, given that JPEG2000 compression software can be designed to efficiently utilize new multi-processor environments, the hard disk reading performance eventually becomes critical. By utilizing the mirrored RAID data storage scheme, in which multiple hard disks are combined together and data content is duplicated to all disks, the reading performance can be effectively multiplied. Thus, we estimate that a JPEG2000 compression application using a quad-core processor is capable of compressing over 100 gigabytes of image data per hour, which is faster than the time it takes to scan a 20 × 20-mm tissue section with a 0.23 μm per pixel resolution using current scanners.⁷ Therefore, it seems that the computational demands of JPEG2000 image compression can be fulfilled with a software-based solution, and there is probably no need for external JPEG2000 processing hardware.

In general, JPEG2000 differs from its predecessor JPEG in many important ways. With regards to virtual microscopy, the most important improvement over JPEG is the possibility to retrieve any part of the image with random spatial accessing. Moreover, due to the nature of the JPEG2000 data structure, different image resolutions (magnification levels in virtual slides) can be retrieved without compromising viewing speed. A single JPEG2000 file can thus contain the entire virtual slide data, and even data from multiple focus layers, when using the JPX file format. It has been extensively documented that when using the average Peak Signal-to-Noise Ratio (PSNR) image quality metric, JPEG2000 outperforms JPEG at all compression ratios.^13,14,25 If the quality assessment is based on human observations, the superiority of JPEG2000 becomes evident at compression ratios of 20:1 or higher.²⁶ In practise, an 8-gigabyte uncompressed 50,000 × 50,000 pixel virtual slide can be readily compressed into a 300-megabyte JPEG2000 image. If the slide contains vast glass background areas, the corresponding file size can be substantially reduced, without compromising the overall image quality of the cells and tissues to be imaged.

Recently, an expansion proposal for the common TIFF file format, entitled BigTIFF,²⁷ has been suggested as a candidate image format for virtual microscopy. It circumvents the 4-gigabyte file size limitation of TIFF, thereby enabling the manipulation of very large images. BigTIFF does not define a compression scheme of its own. It can contain, for example, several JPEG or LZW compressed image code-streams structured in a tile-like fashion. However, by using JPEG compressed BigTIFF, the benefits of JPEG2000 lossy compression are lost. Incorporating a JPEG2000 code-stream within BigTIFF is questionable, since the internal structuring of the JPEG2000 code-stream already enables, among many other advantages, an efficient spatial random access. In addition, JPEG2000 specifies the JPIP protocol for image remote viewing, which has not been described for BigTIFF so far.

Due to its demonstrational nature, the current version of JVS software package does not include any encryption scheme for transmitted image data. However, should a need for encrypted network transmission arise, it is possible to use a standard secure network protocol, such as SSH or TLS, to tunnel the unencrypted JPIP traffic through it. For example, use of the TLS protocol with Virtual Private Networking (VPN) requires setting up a VPN server alongside the JPIP server in the server end, and a VPN client alongside the JPEG2000 viewer in the client end. The VPN client would then act as a proxy for the JPEG2000 viewer, redirecting the viewer’s requests through the network to the VPN server, and from there to the actual JPIP server, and vice versa.

A Model System for JPEG2000-Based Virtual Microscopy

As shown in our results, JPEG2000 already meets the functional needs of virtual microscopy viewing. We have described a model for an easily upgradeable JPEG2000-based virtual microscopy system that utilizes load balancing (Fig. 1). The system consists of a main JPIP server with two network interfaces: a 100-Mbit link for client accesses (via internet or intranet) and a 1-Gbit local area network (LAN) connection for data traffic between main server and sub-server(s). The images are distributed on the hard disks of the sub-servers. The main server includes a database of the image locations within the sub-server hard disks. The database can also store other relevant information (similar to metadata described earlier). The main server delegates incoming client image requests onwards to the sub-servers according to the image location data. An ideally suited sub-server is an inexpensive workstation equipped with three hard disks (one disk for the images, one disk for their backup, and one disk for system software). The backup disk is an independent hard disk, activated only during a nightly backup. Currently available one terabyte hard disks can contain approximately 3,000 to 4,000 virtual slides. As the sub-server hard disks reach their storage capacity, a new sub-server can be easily inserted into the system. Thus, this represents an efficient and inexpensive upscaling strategy that can be done gradually over time. In addition, the modularity of the system allows parts of the system to be offline without affecting the functionality as a whole.

Fig 1.

A model system for JPEG2000-based virtual microscopy.

Future Prospects

Large-scale clinical utilisation of virtual microscopy requires compatibility with the Digital Imaging and Communications in Medicine (DICOM) standard,²⁸ which is currently recognized as the standard specification for image archival in clinical medicine. At the present time, DICOM includes some parts of the JPEG2000 standard, the foremost being Part 1, which is included in Supplement 61 (JPEG 2000 Transfer Syntaxes).²⁹ Recently, Supplement 106 (JPEG 2000 Interactive Protocol)³⁰ introduced the JPIP protocol as a method of delivering the image pixel data apart from the patient data. A joint international effort towards consolidating pathology concepts and virtual slide imaging with DICOM is currently underway by the DICOM working group for pathology imaging (WG-26).³¹ Thus, we assume that the next step towards full implementation of JPEG2000 in clinical pathology and virtual slide imaging requires integration with DICOM.

Conclusion

We conclude that JPEG2000 is a well-suited image format for virtual microscopy. It enables compressing, viewing, and serving the large image files produced by the modern microscope slide scanners. We developed a demonstrational software package for JPEG2000 compression, viewing, and JPIP-based network serving, all suited for the needs of virtual microscopy. The package is freely downloadable and usable for non-commercial purposes.