Digital Microscopy, Image Analysis, and Virtual Slide Repository

Abstract

Advancements in technology and digitization have ushered in novel ways of enhancing tissue-based research via digital microscopy and image analysis. Whole slide imaging scanners enable digitization of histology slides to be stored in virtual slide repositories and to be viewed via computers instead of microscopes. Easier and faster sharing of histologic images for teaching and consultation, improved storage and preservation of quality of stained slides, and annotation of features of interest in the digital slides are just a few of the advantages of this technology. Combined with the development of software for digital image analysis, digital slides further pave the way for the development of tools that extract quantitative data from tissue-based studies. This review introduces digital microscopy and pathology, and addresses technical and scientific considerations in slide scanning, quantitative image analysis, and slide repositories. It also highlights the current state of the technology and factors that need to be taken into account to insure optimal utility, including preanalytical considerations and the importance of involving a pathologist in all major steps along the digital microscopy and pathology workflow.

Introduction

Almost every aspect of research (and life) has been transformed by digitization, and this includes the visualization of tissues and cells on a microscopic scale. This development in pathology was set in motion with the commercialization of digital slide scanners around the year 2000, which enabled whole-slide imaging (WSI) at a relatively high resolution. There are numerous drivers for the growing use of computer screens instead of traditional light microscopes. Easier and faster sharing of histologic images and consultation with colleagues, improved storage of stained slides with maintenance of constant quality, and simple annotation of relevant features in the whole-slide images are just a few of them. Combined with the development of software for image analysis, scanned histology slides further pave the way for the development of tools that extract highly quantitative data from these digitized tissue sections. These points combined make digital pathology and image analysis a valuable tool for translational tissue-based research.

Digital Microscopy

The term digital microscopy (also referred to as virtual or digital pathology) encompasses the digitization of optically scanned histology slides (scanning) and the viewing of these scans via specialized computer software at a resolution similar to conventional microscopy.^1–3

In human and veterinary medicine, digital pathology is being utilized more and more commonly in the diagnostic space, for consulting, collaboration, and research as well as for teaching of medical and veterinary students and residents.⁴ Application of digital pathology as a viable transdisciplinary research and telepathology consultation tool is best exemplified by a panel of veterinary and human pathologists organized across multiple institutions to conduct histopathological comparison of human and natural and/or experimental animal models of cancer utilizing digital slides.^5,6 These cross-disciplinary collaborations illustrate a noteworthy opportunity presented by digital microscopy to promote comparative medicine and translational science. In similar fashion, researchers can collaborate with other researchers or engage and consult with pathologists. With the recent Food and Drug Administration (FDA) clearance of the first digital pathology solution,⁷ a standard has been set for validation of digital microscopy systems, workflows, and workstations, which will further drive the adoption of this technology. At the same time, the interval between the first appearance of slide scanners and the recent regulatory clearance is indicative of how multifaceted the considerations are of how to establish this technology adequately and how to ensure patient safety in the diagnostic setting. Similarly, within the research space, digital microscopy tools need to be utilized in a fashion that generates accurate, high-quality data and reproducible results. When done correctly, quantitative tissue assessment via WSI opens the door to a wealth of tissue-derived read outs deserving of the label “big data.”

Slide Scanning and Viewing

WSI was first developed by Wetzel and Gilbertson in 1999.⁸ The process entails the digitization of entire or selected areas of histology slides. Digitizing histology slides consists of four steps: image acquisition (scanning), storage, editing, and display of images.⁹ Tissue scanners, akin to a traditional microscope, consist of four main components, including a light source, a slide stage, objective lenses, and a high-resolution camera for image capture.^1,10,11 Scanning of cytology slides and other specimens (e.g., plant material) is possible^12,13 but is not the primary focus of the current manuscript.

Digitized slides can be visualized on desktop computers, laptops, tablets, and even smartphones at a range of magnifications.^4,14 Via these screens or displays, slides can be viewed with similar resolution as with a light microscope. In fact, most viewing software solutions aim to mimic the microscope “interface,” allowing the user to jump between fixed magnifications commonly used as microscope objectives. Additionally, users can zoom in and out on a sliding scale that is akin to navigating Google Maps. In fact, the technologies that drive WSI image viewing are very similar to those of major geospatial systems, such as Google Maps. These images contain substantial amounts of data and, unlike other images, surpass the computer’s memory capacity. To mitigate this challenge, part of images (“tiles”) are opened from the image and fed to the image viewer for reassembly rather than the entirety of the whole-slide images.

Navigating a slide in a 2-dimensional fashion (x- and y-axis) is effortless via the use of a computer mouse. Some scanners capture several scans along the z-axis and assemble them on top of each other (“z stacking”) to allow digital fine focusing on an area of interest.^3,4 This can be especially helpful when reviewing cytology preparations or identifying microorganisms.^1,2,15

Slides scanners acquire images of histology sections either in tiles or linear patterns (Figure 1). The multiple images are rapidly captured and digitally assembled (“stitched”) in real time to create the image of the entire slide.^2,14 Modern scanners can obtain slides stained with H&E, special stains, immunohistochemistry stains (bright field), and fluorescence stains. Bright field imaging recapitulates standard bright field microscopy, representing the most commonly used and most cost-effective approach. Fluorescent scanning recapitulates fluorescent microscopy and is used to digitize fluorescent immunohistochemistry (IHC) and in-situ hybridization (ISH) slides.¹⁴ Multispectral imaging (MSI) can be performed in both the bright field and fluorescent setting, and captures spectral information across the visible range of light up to or near infrared.^16,8 MSI allows the simultaneous imaging and quantitation of multiple markers even in the presence of spatial and spectral overlap.¹⁷ In immunofluorescence, MSI is specifically suited to overcome issues due to tissue autofluorescence, which is far brighter in formalin-fixed tissues.^8,14 This is achieved because the spectrum of each individual marker is unmixed, which separates the parts of the spectrum so that those wavelengths of interest can be isolated. This imaging technique can be performed with commercial imaging instruments suited for multispectral imaging or the combination of a standard fluorescent microscope in combination with a multispectral camera (and generation of a multispectral library of each fluorophore used).^17,18 Mansfield provided an in-depth review of multispectral imaging technology and its use in anatomic pathology.¹⁷

Figure 1.

Scanning patterns. (A) Tile scanning of every tile. The arrows indicate direction of scanning (modified after Indu et al. 2016).¹⁴ Dots within a tile indicate a focus point. (B) Tile scanning of every nth field. (C) Line scanning pattern. Dots indicate focus points of a focus map.

Independent of the scanning and focusing methodology, most scanners allow for scanning at multiple magnifications. Most commonly, this includes 20× and 40× magnification. Scans at 20× magnification are sufficient for standard viewing and interpretation and even most image analysis of H&E and IHC slides. However, scans of ISH slides should be performed at 40× to allow for better resolution of single dots. Scanning at higher resolution (60×/63× or 100× magnification, under oil) is now available for select scanners and is recommended for blood smears and similar preparations.^3,4,10 Only a few scanner models are available that can accommodate both dry scanning and oil immersion.¹ It is important to note that the higher the resolution used for scanning, the larger the data file created for each slide, which then needs to be archived and stored.² Doubling the magnification from a 20× scan to 40× increases the file size by approximately 4 times.^4,19 In addition, capturing sections in several z planes further contributes to large file sizes.⁴ Depending on the downstream use of the scan, one needs to be cautious with compressing images to reduce file sizes. While research has shown that image compression has no to little effect on the diagnostic results of a skilled observer, image analysis results can be impacted by loss of information due to compression.^2,20 Morphological assessments via algorithms are less affected, but densidometric assessments are more sensitive to this loss of digital information.² The current standard for compression in WSI is the JPEG2000 compression scheme, which is based on discrete wavelet transforms. JPEG2000 allows the highest compression-to-quality ratio and is preferred over typical JPEG compression.

The capacity of scanners and their scan times can vary between scanning modalities and scanner models. Higher throughput scanners now have loading capabilities to accommodate up to 400 slides,^1,14 with scanning times ranging from less than 30 seconds up to several minutes per slide.^1,2,21 Scanning larger tissue sections, and/or at higher resolution, not only increases the file size, as previously mentioned, but also lengthens the overall scanning time.⁴ Specialized scanners that can digitize whole tissue mounts and larger glass slide sizes are available.¹

Digital slide scanners are being marketed by multiple vendors. Reviews of the different options have been published elsewhere,^1,22 but more recent reviews are sparse and new models and technology are constantly released, rendering any publication quickly outdated. This continual parallel advancement has also led to a plethora of WSI file formats, some being proprietary. Although adoption of a single open-source image file format standard has yet to occur, open-source software libraries such as OpenSlide from Carnegie Mellon University exist that have helped ease the differentiation between vendors by abstracting the reading of image data through a single entry point.^23,24

While most academic research is conducted in a nonclinical environment, basic standards of good laboratory practices also apply to digital microscopy workflows, equipment qualification, calibration, and maintenance.¹¹ Guidelines have been published on establishing WSI systems in regulated nonclinical environments.²⁵ The extent to which these guidelines are applicable to any nonclinical laboratory that is not focused on safety assessments of test articles may vary from case to case.

Digital Workflow

The workflow of digital microscopy differs from the traditional histology workflow in that it contains additional steps for digitizing histology slides after specimen-slide preparation and staining.²⁶ This includes additional equipment and staff, properly trained personnel, appropriate quality control steps, continuous maintenance of machines and software packages, and adequate information technology (IT) infrastructure.

In addition, proper workstation set up for viewing and analyzing digital slides has to be considered.²⁷ While it is generally accepted that the quality of a digital slide heavily relies on the quality of the physical slides,^15,21,28 it is also important to consider that displaying scans on a computer monitor that is inadequate in terms of resolution and color display has the potential of impacting interpretation and the data generated.^15,29 This is especially the case when digital slides from the same study or experiment are viewed on several different monitors. However, discrepancies in color perception and viewing of grey scales by humans evaluating whole-slide images on computer monitors may have a similar, if not greater, impact.^29–31 While color calibration describes the process of matching color displays between different units (e.g., monitors), end-to-end color calibration ensures that the color stays consistent throughout the entire workflow. In the context of digital pathology, this means from tissue staining to scanning and displaying the color on a monitor.²⁹ Both aspects need to be factored in when establishing a digital microscopy workflow.

In addition to the color display of a monitor, screen size and resolution are important for adequate viewing of digital slides.^32,33 Utilizing a monitor at a resolution that is lower than the WSI resolution results in loss of detail at higher magnification and an overall more pixelated view.²¹ A color-calibrated monitor with full high definition (1920 × 1080 pixel dimension) should be considered a minimum requirement for assessing detailed scans.²¹ One can purchase a monitor that was calibrated at manufacturing or utilize software solutions to calibrate monitors post purchase.³⁴

If the computer that is used for reviewing slides does not have a local copy of the scan or is not directly linked to the server hosting the images, then internet bandwidth and latency can have an effect on the time needed for focusing a view, changing magnification, and moving around the slide.²¹ Minimum internet connectivity speeds of 20 to 100 megabits/s have been recommended.^15,33,35

Another important part of the digital pathology and image analysis workstation is the tool used to navigate around a slide. A generic computer mouse is likely the most commonly used tool; however, more sophisticated devices such as touch screens, Wacom boards, and three-dimensional (3D) space navigators are available for more ergonomic, frequent use and easier maneuvering. The selection of such tool appears to be in part personal preference.

While viewing angle and surrounding light can influence perception of bright field scanned images, it is paramount to have a more controlled and consistent viewing environment when evaluating fluorescent scans.

Role of the Pathologist in Digital Microscopy Workflow

Along the entire process from harvesting fresh tissues to generating a stained slide, every step involved can have a significant effect on the quality of the final slide. These so-called preanalytical variables profoundly impact tissue morphology and stain quality, therefore determining how amenable a slide is to interrogation via digital image analysis and driving the validity of the generated data.^28,36,37 Examples are as obvious as effects of improper physical handling of unfixed tissue generating disrupted cellular morphology (e.g., crush artifacts) and as subtle as the influence of water bath temperature on the expansion or shrinkage of individual thin tissue sections. The pathologist is uniquely qualified to evaluate the impact of preanalytical variables on generated data because of their knowledge of all aspects of this process. Therefore, pathologist input should be central to all major steps along the digital pathology workflow, beginning with study design, continuing through sample quality assessment, tissue staining, scan quality, guiding analysis parameters and design of data capture, final data interpretation, and reporting.²⁸

In more detail, before slides should be subjected to digitization, an experienced pathologist must evaluate whether the specimens are of sufficient quality and confirm that the target tissue is present in adequate quantity. Within an individual study, it is important to standardize tissue harvest and fixation methods, fixation time, and staining protocols to minimize the heterogeneous effects of preanalytical variables.^38,39

In addition to the act of performing tissue staining on study slides, the prior optimization is paramount in generating meaningful research data. A sufficiently optimized staining protocol not only adequately addresses the presence or absence of a biomarker but also provides the opportunity to answer more detailed questions, including presence of the biomarker in subcompartments of cells, differentiating various expression levels, and, at times, colocalization with other markers of interest. Important parts of creating consistent staining results are the use of automated staining platforms and the uniform use of appropriate on-run assay controls.^28,39 Especially for IHC staining, a pathologist’s confirmation that the staining occurs in the correct cellular compartment, and in absence of nontarget tissue staining, is important. In addition, it is vital to verify that the staining intensity covers a dynamic range to allow for differential assessment of staining levels.²⁸ Specifically, chromogenic IHC does have a lower and nonlinear dynamic range compared to fluorescence IHC.⁴⁰ More detailed information about various staining techniques, including immunofluorescence IHC and ISH, is reviewed elsewhere.⁴¹

After digitization of slides, the scanning quality needs to be confirmed as well. Scanning artifacts can affect downstream results and include improper cleaning of slides prior to scanning, poorly focused scans, and compensation lines from improper stitching of lines or tiles (reviewed elsewhere).²⁸

If data generation occurs by a nonpathologist, final review of the data by a pathologist to ensure adequate quality is advised. In addition, interpretation of the data in the context of the research question or disease biology should involve disease process (pathology) expertise.²⁸ Unfortunately, many examples exist in the biomedical literature where normal histological features were misdiagnosed as diseased, for example, the misdiagnosis of mouse nipples as “premalignant” papillomas (a number of examples are included in the cited publications).^38,42 In absence of direct access to pathology expertise, alternative ways of selecting and engaging a pathologist as a collaborator or consultant could be considered, including by digital telepathology. Many academic or research institutions have pathologists on staff, albeit the headcount may be low and positions are likely outside of a specific pathology department or unit. Institutions that are not engaged in training of medical or veterinary professionals may have pathologists as research faculty or staff pathologists to provide pathology support via a core facility.

If a pathologist is not available from within the research organization, researchers may contact the various professional organizations that govern the pathology profession to request recommendations of members with expertise in the specific area of study. These organizations include but are not limited to: (American or European) College of Veterinary Pathologists, College of American Pathologists, Society of Toxicologic Pathology (main organization in America, with local organization in Europe, Britain, and Japan), American Society of Investigational Pathology, and Digital Pathology Association.

Digital slides greatly expand access to pathology resources. Many pathologists are experienced and comfortable using WSI and consulting over the internet. Some research organizations support regular telepathology meetings featuring live, online discussions based on WSI.

When engaging with pathologists or researchers at other institutions, it is important to be aware of both institutions’ IT policies, specifically with regards to firewall, data integrity, and security. If these aspects present a hurdle, it is advised to engage members of the institutions’ IT teams to jointly work on practical solutions (e.g., shipping encrypted external hard drives, etc.).

Digital Slides as Collaboration and Research Tool

Among the advantages of digital microscopy in the research setting is that scanned slides can be shared with several collaborators without the risk of loss or breakage.¹² In addition to sharing an entire tissue image, annotations can be communicated to highlight particular areas of interest.¹⁵ When multiple observers are involved, it is noteworthy that everybody reviews or works with the exact same slide image instead of recuts that may or may not show similar tissue features. When a digital microscopy infrastructure is in place, sharing images or slides with multiple international institutions can be much quicker, easier, and potentially cheaper than shipping fragile glass slides all over the world.^1,12 One can reach an expert anywhere on the planet as long as that person has a computer and an internet connection—no microscope needed.¹² Just as importantly, digitized slides allow for virtual meetings and review of specimens with multiple experts at the same time.⁴³ Impressively, one of the authors (R.D.C.) has collaborated with scientists in 18 different countries across five continents using the internet and digital pathology. In addition, since only one good-quality slide needs to be produced for scanning and sharing, exotic findings and small tissue samples are no longer a limitation.¹

For the individual researcher who is reviewing digital slides, one significant advantage is the option of viewing several scans of the same tissue or related specimen side-by-side.³ This can be, for example, liver slides with the same stain from different mice of the same study, or reviewing H&E and multiple IHC stains for different markers on the same tissue sample all next to each other (Figure 2). Some advanced interfaces allow synchronized viewing options that enable simultaneous and coordinated exploration of multiple slides.³ Within the work of phenotyping genetically engineered mice, viewing whole embryo or neonate sections of wild-type and genetically modified animals side-by-side is especially useful to detect and annotate developmental abnormalities. In this regard, side-by-side viewing of whole embryos is particularly valuable to ascertain whether sections are of comparable orientation, a critical consideration in embryo or neonatal histoanatomical phenotyping. Lastly, viewing and scoring tissue microarray (TMA) slides digitally is usually easier, as it is more convenient to assess and track the location of each core within the grid tissue array.^2,15 Specialized software applications that facilitate seamless TMA navigation and review are available by multiple vendors.

Figure 2.

Assessing multiple characteristics of the same specimen simultaneously in digital slide view. A display of digital slide viewer depicting side-by-side viewing of H&E stain (A) and different immunohistochemical markers (B–D) on serial sections of a mouse prostate tumor. Primary antibody chromogenic detection with DAB and hematoxylin counterstain (B–D). Inset rectangle at the bottom right of each of the four displays shows a full slide overview; the small rectangle within each inset (arrow example) shows the region that is displayed at higher power magnification (10× in this case). Glass slides were scanned at 40× magnification with NanoZoomer slide scanner and viewed with NDP viewing software (Hamamatsu Photonics, Bridgewater NJ, USA).

High-resolution scans of good quality histology sections enable easy image capture and yield superior quality images for publication figures and potentially journal cover art. In addition, some image software solutions feature figure-making options that allow generation of digital image panels that can be incorporated in reports and publications with relative ease. More recently, some journals give online access to digitized slides pertinent to a publication as supplemental material or as a modern interpretation of a figure in a manuscript (example cited here).⁴⁴

Of course, there are still hurdles to overcome in the widespread adoption of digital microscopy, which include the investment for equipment, expanded laboratory footprint, training of staff, and IT infrastructure. As a result, imaging core facilities have been established at many academic institutions to serve as one central resource for a multitude of researchers on campus.² Besides these financial limitations, researchers and pathologists alike have at times been reluctant to embrace the new technology, likely from habituation to traditional microscopy.^4,28 Another important barrier to adoption of digital pathology is the lack of awareness of the added value of this technology in generating information or data above and beyond routine light microscopy. Recent progress in digital microscopy demonstrated the capacity and opportunity to explore and mine “subvisual” image features from digital slide images that may not be visually discernible by a pathologist, hence offering the opportunity for better quantitative modeling of disease appearance and improved prediction of disease aggressiveness and patient outcome.⁴⁵ Furthermore, disease processes and severity, as well as changes in biomarker expression, usually span a continuous scale and cannot be adequately interrogated or captured by binary research approaches such as presence or absence of a lesion and/or a biomarker.³⁷

Quantitative Image Analysis

Basics of Image Analysis Approaches and Available Tools

Digital image analysis, also called tissue image analysis or quantitative image analysis, in the context of histology specimen encompasses the extraction of data from digitized histology sections via analysis algorithms in an automated image analysis approach. There are a multitude of different solutions and algorithm approaches available, and the number and functionality of applications appears to be steadily increasing.

In broad strokes, tissue image analysis has three major areas of application: area-based measurements, cell-based measurements, or measurements pertaining to other objects in the tissue aside from single cells.^28,46 Furthermore, to perform these types of analyses, there are open-source tools available,⁴ commercial software packages for purchase, and service providers who will perform analysis on a fee-for-service basis.²⁸ In addition, numerous research institutions and academic laboratories have generated their own custom-made analysis algorithms or built upon open-source tools to create algorithm solutions that fit a variety of their specific needs. However, for anybody working with these tools, it is important to know the capabilities and limitations of the specific systems that are being utilized.²⁸ Most importantly, the reproducibility of automated analysis should not be confused with reliability or accuracy.⁴

In general, area-based measurements assess tissues based on color or color intensity of a particular stain and quantify the surface area (or number of pixels) of the slide that contains this set color and/or intensity. For cell-based assessments, pixels are compared based on similarity in features and grouped together to identify structures that are comprised of several pixels, such as nuclei. Identification of nuclei is the most commonly used stepping stone to identify entire cells. However, there are ways to identify and define cells that do not rely on nuclear detection first. Once nuclei or cells are defined, their size, shape, staining characteristics, and spatial relationship to each other (e.g., close together or farther apart) can aid in identifying tissue compartments that are comprised of cells. For example, algorithms can be developed to distinguish between neoplastic cells and surrounding stroma, identify multicellular structures such as glomeruli or follicles from any other tissue compartment, or separate inflammatory infiltrate from resident tissue. This, in turn, enables the analysis of cellular features within the identified tissue compartment. Furthermore, detecting the individual cells within a tissue also enables a researcher to focus the analysis to specific cellular subunits, for example the quantification of ISH dots exclusively within the nucleus or cytoplasm of the cell population of interest.²⁸

Similarly, object-based algorithm assessments identify specific tissue objects, for example, vessels. Such algorithms can be used to assess and quantify each object in its entirety instead of cell-by-cell data extraction from the object. For example, while blood vessels are comprised of multiple cells, an algorithm specific for the analysis of vessels may not focus on each cellular unit but extracts data for each vascular profile/cross-section, including cell-independent measurements such as vessel circumference, diameter, lumen, or roundness. In addition, object-based measurements do not have to exclusively focus on objects that are comprised of cells (e.g., analysis of amyloid plaques).²⁸

A thorough review of the various software solutions is beyond the scope of this manuscript; however, other publications are available.^22,47 Ultimately, a decision oftentimes is based on factors such as in-house software development expertise, timelines, budget, previous knowledge of image analysis tools, and anticipated frequency of use.

Most recently, the increase of constantly improved solutions for quantitative automated image analysis is mostly based on the successful integration of deep learning techniques into visual pattern recognition. It is important to note that deep learning is considered a subfield of machine learning and that it is not a single technique, but the application of a principal approach focused on feature learning.^48,49 Via this methodology, a researcher may develop or utilize well-established architectures (such as AlexNet or Inception) and frameworks (such as TensorFlow and Theano) to create software that will generate statistical models for the identification and classification of features. In the supervised learning approach, for example, this means that thousands or millions of images of a training data set are fed via convoluting layers as feature inputs into a neural network so that it learns to recognize these structures autonomously in unknown images.⁵⁰ The algorithm does so by freely selecting and extracting the most important morphological features of the structure of interest, a layered process called “convolution.” This autonomous learning is currently mostly performed by deep artificial neural networks in the so-called deep learning approach.^51,52 Artificial neural networks consist of numerous, connected computing units, the “neurons,” that are arranged in cascading layers (Figure 3).^49,53 Entering these layers, the input data are transformed and passed on through weighted connections and biases to the next layer, a process called “forward propagation.” Once the network has been propagated through, the error is calculated from a loss function and the parameters are readjusted (“back propagation”) and the forward propagation is started over again. This process is iterated over thousands of training cycles (or “epochs”), as the software learns autonomously to separate the structures of interest in the training images.⁵² For instance, a neuronal network trained with 10,000 examples of mitotic cells and diverse nonmitotic cells was able to identify mitotic cells in tumors with an accuracy of 94%, a value that is similar to what is comparable to manual counts performed by trained pathologists.⁵⁰ Another remarkable demonstration of the application of neural networks is the classification of neoplastic skin lesions.⁵⁴ In this study, deep convolutional neural networks executed binary classification of most common skin cancers (keratinocyte carcinomas vs. benign seborrheic keratoses) and identification of melanoma from benign nevi with comparable performance to that of a panel of board-certified dermatologists.⁵⁴ Some of the models obtained by neural networks are then provided as finalized solutions in commercial image analysis software packages, for example, as standard applications for automated cell or nuclear detection. Other software solutions even provide embedded simplified machine-learning functions that allow the user to establish individual problem-specific image analysis solutions that may be able to identify and quantify broader tissue characteristics, such as automatic detection of necrotic tissue.⁵⁵ Comprehensive reviews of deep learning in medical imaging have recently been published.^56,57

Figure 3.

Simplified schematic of deep learning approach. Example images of epithelial cells and other cells are fed into a convolution layer that passes on features to a neural network to learn to identify epithelial cells autonomously in unknown images. As the information is passing through the neural network (comprised of connected computing units called “neurons” arranged in layers), the algorithm freely selects the most important image features of epithelial cells. Entering these layers of neurons, this feature information is transformed and passed on through weighted connections to the next layer. The algorithm completes thousands of training cycles to learn to recognize these structures autonomously in unknown images.

Role of Pathologist in Data Generation and Review

As mentioned previously, when evaluating histology slides and generating tissue-based data, a pathologist is uniquely qualified to oversee the data generation not only because of training in all the processes that influence the appearance of the tissue of interest on the slide but also because of the combination of general pathology training and extensive knowledge in disease biology, model organism, histology, and histopathology.

While it is easy these days to get lost in the allure of big data and complex statistical analysis, prior to further analyzing data generated from digital image analysis it is of the uttermost importance that a pathologist is involved in the review of the algorithm performance. Although digital image analysis produces results with a higher level of reproducibility and consistency, its accuracy in certain aspects needs to be confirmed by a pathologist.⁴ In other words: Is the algorithm truly measuring what was intended to be measured? Is it identifying the right areas, tissue structures, and cells? And when those aspects are analyzed, is the generated data biologically relevant and meaningful?²⁸ While in a regulated environment, the review by a pathologist is required by the FDA⁵⁸; it is strongly advised that nonclinical laboratories and research projects incorporate a similar review step in their image analysis workflow. Often, the performance of an algorithm is tested by comparing to manual pathology results, for example, manual IHC scores of a TMA compared to algorithm-generated scores to qualify biomarker expression analysis.³⁰ However, even when concordance with manual scores is achieved, this does not automatically mean that the algorithm is correctly measuring the intended cells and selected parameters. It is further insufficient to imply that because results from a test set matched up, that the algorithm solution will perform as intended when applied to new samples, without a continuously utilized review step by a pathologist. Depending on the context of the study, a review of every algorithm applied to each slide is oftentimes necessary and at times specifically required (e.g., in a regulated environment).^28,30

Limitations of Semiquantitative Pathology Scoring

Currently, morphologic lesions in laboratory animals are mostly quantified manually by the pathologist. However, at times, the phenomenon intended to be quantified may be too complex or too subtle to be reliably assessed by eye and interpreted by a human brain, for example, the exact measurement of affected area or the exact enumeration of a particular cell population.^2,4,28 While digital image analysis opens the door to extract exciting and novel data points from tissue sections and antibody-based stains, it is also a powerful tool that can enhance these routine and long-established methods such as manual pathology scoring and grading.^7,59 Usually, manual scoring, for example of IHC staining to assess biomarker content and distribution, generates data on an ordinal number scale, or derivative thereof, and therefore is (1) not a quantitative measurement of ground truth, and (2) is not amenable to typical statistical analysis.^60,61 Digital image analysis tools can aid in the generation of data points on a continuous scale that are available for stringent statistical interrogation.⁶² In addition, these tools can help create more reproducible and repeatable results.^2,63

In general, histopathological scoring and grading enables the pathologist to extract qualitative or semiquantitative data from tissue sections.⁶¹ Therefore, any scoring or grading paradigm employed should be definable and reproducible, and produce meaningful results.^61,64 Prior to data collection, the details of any paradigm utilized should be defined and documented, independent of using a well-established or study-specific and customized scheme.⁶⁵ Scoring paradigms and concepts pertinent to manual assessment of IHC staining have been described and reviewed elsewhere.^30,61,66 It has often been observed that when comparing manual scores or grades generated by several pathologists within the same sample set, the different data points identify the same overall trend, although oftentimes inter-observer variability is substantial.⁶⁷ This variability may be due to multiple factors. At times, grading schemes and scoring paradigms are complex and/or ambiguous and therefore challenging to apply in a consistent manner. At other times, the observer is the limiting factor, being influenced by bias or biological shortcomings. For example, the human perception of staining intensity and color hue is inherently variable, and this can influence the interpretation of tissue sections. Biases, visual limitations, and cognitive traps that may influence assessment of tissue slides have been extensively reviewed elsewhere.³⁰ Some of these effects are a human phenomenon that apply to any observer, trained pathologist or not, whereas other biases may create less of a challenge to the skilled observer.³⁰ Digital image analysis tools, if used adequately, can be a valuable aid in avoiding these pitfalls. Analyses employing algorithms are not affected by biases such as number heaping (also knowns as number preference) or avoidance of extreme ranges. Staining intensity, color, and hue are captured digitally and assessed by computational algorithms that are far superior to the human nervous system in identifying and quantifying color. Just as important, algorithms enable researchers to consistently apply the same evaluation metrics slide after slide and thereby increasing repeatability and reproducibility. In addition, computational tools are able to treat every slide or algorithm run as an independent event. In contrast, it has been shown that pathologists assessing the severity of a lesion in a particular tissue section of a sample set are influenced by the severity score of up to five slides viewed immediately prior to the current slide, demonstrating the challenges of evaluating every slide independently.⁶⁸ Data generated via image analysis may also aid in generating classification for parameters that are challenging to assess accurately by eye, such as scoring tumor heterogeneity.^2,69,70 Aside from improving intra-observer variability, digital image analysis tools also aid in improving inter-observer concordance, for example, by preventing diagnostic drift.⁷¹ However, algorithms currently cannot replace the human observer, and greatest accuracy will be achieved when input images optimized for analysis are evaluated using algorithms and data outputs assured by pathologists and others trained in computer-assisted diagnostics.³⁰ This is not only due to the pathologist’s ability to review algorithm markups and identified “failed” algorithms but also to algorithms’ inability to identify unique lesions and the need for manual pathology input when analyzing specific areas of interest (i.e., tumor margin vs. tumor center). Last but not least, digital image analysis tools may aid in higher throughput systems and screening, such as semiautomated assessment of tissue microarrays.⁷²

In summary, the synergy of an experienced pathologist, with the precision and consistency of an optimized algorithm, will lead to more robust and reproducible data that are amenable to rigorous statistical analysis.^28,62,63

Prospects and Examples of Digital Image Analysis in Translational and Biomedical Research

Digital image analysis tools can be applied to a variety of different tissue preparations, including formalin-fixed, paraffin-embedded tissues; cryo-sections; and whole embryo or organ mounts.^28,73,74 Similarly, algorithms can be applied to routine H&E-stained sections, histochemical stains, chromogenic IHC stains, fluorescent labeling, and ISH stains.^46,75 A number of examples are displayed in Figure 4.

Figure 4.

Examples of utilization of digital image analysis as research tools. (A) Example of area-based assessment in quantification of cardiac fibrosis (⁴⁴). In the markup image, blue areas are those of cardiac fibrosis, red areas are those comprised of cardiomyocytes, and yellow is empty slide area. An algorithm can extract exact quantification of the size of each surface area. (B) Example of utilization of nuclear orientation in the identification of different glandular structures (⁷⁶). The lines overlaying the nuclei are generated by an algorithm and represent the axis of each nucleus along its major diameter (largest width). Analysis of the relationship of these axes to each other enabled researchers to separate glandular structures (more parallel axis orientation) from other tissue components. (C) Example of cell-based assessment in the separation of stromal cells (middle panel) and epithelial cells (right panel) to quantify positive cells based upon nuclear IHC signal (⁷⁵). Extraction of counts for each tissue compartment is possible via algorithm but also of morphological features (e.g., size, roundness, major diameter, etc.) for every individual cell analyzed here. Green cells = not selected; blue cells = target cell population, IHC negative; red cells = target cell population, biomarker positive.

Computational algorithms can aid in the identification and separation of different tissue types. In a noteworthy demonstration of the potential of these tools in classification and separation of diverse tissues types, automated pattern recognition image analysis was utilized on morphologically complex teratomas. These were tested as unknown specimens not used in training the algorithm and showed diagnostic agreement with pathologist assessments of stem cell pluripotency in a subset of samples.⁷⁷ Similarly, automated image analysis has also been utilized to quantify metastatic burden in a mouse model of lung metastasis.⁷⁸ When performing “simple” area-based assessments, algorithms have been shown to outperform manual pathology evaluations, for example, in the quantification of myocardial fibrosis (Figure 4a).⁴⁴

Within a particular tissue type, algorithms can be developed that reliably identify specific features, such as the separation of luminal from glandular epithelium in uterus samples (Figure 4b). This was achieved by analyzing the orientation of the cells and their spatial relationship to each other.⁷⁶ Identification of such structure subsequently renders them amenable to a more in-depth analysis of morphological features. For example, recent published work utilized digital image analysis for morphological characterization of mouse mammary glands. In this work, mathematical and morphometric concepts from studying angiogenesis of blood and lymph vessels were expanded and applied to the developing mammary gland.⁷³

There are numerous examples in the literature that demonstrate the use of digital image analysis algorithms to aid in the identification of lesions with subsequent refinement of scoring paradigms, for example, in the study of murine colitis models, porcine models of radiofrequency ablation of metastatic lesions, or the assessment of intratumoral heterogeneity in mouse models of human breast cancer as well as in native patient samples.^55,69,79 Similarly, the use of digital image analysis tools has demonstrated improved data quality and robustness as in the evaluation of liver fibrosis and hepatic steatosis in animal models.^71,80

Especially in the quantification of immune infiltrates in tissue, digital image analysis tools can be valuable, as manual scoring paradigms are challenging. The approach often requires multiplex staining and can be used in identification of lymphocyte subpopulations, which has become increasingly popular with the interest in immunooncology. As an added benefit, in the study of melanoma samples, multispectral imaging is a valuable tool to overcome the added hurdle of pigmentation of cells, and digital image analysis is a reliable tool to extract data from this imaging modality.^16,81 Lloyd et al. reviewed a variety of area-based image analysis assessments utilized in the interrogation of the tumor microenvironment.⁸²

Digital image analysis has also been used to quantify activation and cellular localization of signaling effector molecules in a mouse model of lymphoma and melanoma; in the latter work, digital image analysis was used to assess quantifiable differences in on-target drug response in experimental oncology.^83,84

At times, an algorithm can be a tool to simply avoid tedious and tiring tasks such as manually counting biomarker positive cells by pathologists, as demonstrated in enumeration of Ki67-positive and/or immune/inflammatory cells in a genetically engineered mouse model of cancer or, more impressively, the identification and counting of mitotic figures (Figure 4c).^50,75 Similarly, in the field of ISH dot quantification, digital approaches have demonstrated more accurate performance than visual assessment by pathologists.⁸⁵ To assess added complexity, digital approaches can also be used to investigate colocalization of ISH dots and IHC staining to interrogate the relationship of RNA expression and protein signal of the same biomarker in individual cells.^86–88

One of the potential applications of digital microscopy in the immediate future may be in the context of the discovery of microscopic morphological phenotypes in mutant mice. Data from large-scale mouse phenotyping centers show that genetically engineered mice often lack correlative microscopic findings despite presence of clinical phenotypes on a battery of phenotyping tests.^89,90 Lack of correlative microscopic lesions may be attributed to the fact that routine light microscopy might not be sensitive enough to detect subtle but biologically consequential morphological differences between wild-type and mutant mice. This is likely the case when changes are associated with variation in cell numbers. It is estimated that, depending on the tissue, the magnitude of changes in cell number must reach 25% to 40% before it can be visually appreciated by the pathologist.⁹¹ These observations argue that more sensitive quantification methods may be needed to detect subtle morphological phenotypes in mutant mice or other in vivo models in absence of obvious microscopic findings. Digital microscopy may therefore complement routine histopathology in large-scale mouse phenotyping efforts, especially in mutant mice with clinical abnormalities such as neurological deficits that typically are not associated with obvious histopathological correlates.

Stereology

It is important to remember that while WSI allows for digital image analysis of an entire tissue section, it is still only a two-dimensional slice of a 3D object.³⁷ Inferring events (i.e., cells, IHC staining components, vascular profiles) in a 3D tissue piece based on observations in a two-dimensional tissue section is common practice and in fact represents the vast majority of analyses. However, when transforming data from the digital analysis of tissue section into a quantitative output, the data are biased and at times inaccurate.^92–94 Importantly, irrespective of the manner used to evaluate the number of events in a tissue section, either by eye or digitally, the final data still remain an estimate. While these can be precise, per definition, they are never completely accurate. This results from the fact that routine tissue sampling and evaluation of histology sections are inherently biased (as previously described). For example, how a profile of an object (e.g., cell) appears within a tissue section is influenced by the object’s size, shape, orientation, and distribution, is known as geometric bias. Because the enumeration of objects in a tissue section is in fact largely influenced by the object’s size and orientation, enumeration of these objects based on individual tissue sections is inherently biased.⁹¹

In contrast, stereology utilizes techniques including stringent sampling methods to obtain 3D information about the entire tissue in an unbiased fashion, ultimately generating results that are absolute numbers/values instead of density estimates and ratios.⁹⁵ In addition, when sampling sections throughout the third dimension of a tissue, stereological principles can be used to factor in tissue shrinkage during processing, which can be variable across tissues, treatment groups, etc.^94,96 However, since design-based stereology is not only time and resource intensive but also requires the usage of a significant amount of tissue, it is often not considered a routine methodology. Notwithstanding, the advancement of digital image analysis solutions has improved the efficiency of stereological tissue evaluation, and commercially available solutions specifically for this approach are available.

An additional limitation of stereological approaches is the fact that they cannot be employed retrospectively but require extensive and detailed upfront experimental planning, which often heavily relies on the results of a pilot study.⁹⁷ Involving a skilled stereologist (e.g., a pathologist with extensive stereology experience) is paramount to successful study design, data generation, and interpretation.

Design-based stereology should be considered when an analysis may fall below the range of detection by a standard light microscopy survey or an unbiased estimate of effect level is desired to assist in determining the magnitude of difference between various experimental groups. Specifically, if power analysis determines a small magnitude of difference given all variance (i.e., <5×), design-based stereology should be considered a requirement for meaningful data generation. In addition, it is important to note that the inherent bias of image analysis is recognized and regulated by a number of high-impact journals.^98–101

For extensive background and statistical methodologies regarding design-based stereology, the reader is encouraged to study the introductions to the discipline provided by Hans Jorgen Gundersen, Rogely W. Boyce, and other stereology experts.^96,102–105

Slide Repositories and Metadata Retention

Storage of glass histology slides requires significant amounts of space; large numbers of glass slides results in a substantial amount of physical weight, which needs to be taken into consideration when evaluating adequate storage systems. Furthermore, filing and retrieval of slides is time-consuming and error prone. In contrast, digitized slides do not take up physical space and are easily accessed on demand through digital storage systems. However, adequate and functional digital storage in a searchable system is critical to creating a meaningful database. There is a requirement of a minimum database for adequate digital slide storage in terms of specimen-specific information, so that slides may be rapidly accessed and a searchable database is maintained. Without specific information related to study, animal and group number, species, sex, morphologic diagnosis, etc., an optimal searchable system cannot be established. In addition, management of these systems in terms of accessibility, maintenance of data, and data security poses significant challenges, often necessitating additional dedicated employee positions to handle these responsibilities. WSI of certain specimen is time-sensitive, or requires attention to specific protocols. Specifically, in the realm of fluorescent staining modalities, adequate storage and indexing of slide scannes is crucial, as the physical slide’s staining will significantly fade within a short period of time, even under optimal storage conditions.⁴

The opportunities of accessing digitized slides in searchable databases or repositories are manyfold. First, slides can be immediately retrieved and shared or discussed with others on demand for teaching, consultation, or performing studies, including when new research information becomes available that may prompt re-review of older specimens. In general, searching a digital database is less time-consuming than retrieving physical slides. Compared to glass slides, digitized slides are stable in quality in contrast to fading, breaking, or loss of cover slips often encountered with glass slides. Remote access to digitized slides stored within such database can be established, enabling extensive collaborations past the physical address of researchers. In contrast, if consultation or collaboration is needed between researchers at different institutions, the alternative with physical slides requires shipping of often large numbers of glass slides, which is time-consuming, costly, and comes with the risk of potential loss or damage of slides. Depending upon the database, digital slides can be linked to relevant documents, such as a study plan or a research report, to facilitate quick access to all materials pertaining to a given study.^3,4,27,106

Depending upon the volume of scanned slides and the size of the laboratory, storage solutions and backup mechanisms may vary in size and complexity. Large diagnostic laboratories that create and scan more than 13,000 slides per month likely have a larger IT and database structure with complex backup and retrieval procedures.²⁷ However, even a small department that runs a slide scanner or a research laboratory that scans relatively small number of slides should spend sufficient time on planning and maintaining a digital archiving system that allows retrieving digitized slides via a searchable database.

Most commercially available scanners come with access to a digital platform that hosts slides and makes them available for viewing via an internet connection. However, currently most vendors promote their own platform and file format, often with minimal ability to store and view other file formats. Standardization within this space is still lacking, but efforts are underway. Within the field of digital radiology, the Digital Imaging and Communications in Medicine (commonly referred to as DICOM) standard has been established and successfully implemented.¹⁰⁷ This standard requires some modifications to be applicable to digitized slides, and efforts are ongoing.¹⁰⁸

Instead of occupying physical space, digitized slides occupy digital storage space. While in some settings this may require a significant investment in IT infrastructure, the cost of digitized memory is predicted to further decrease according to Moore’s law.¹⁰⁹ In addition, optimal compression can influence the amount of digital storage needed without reducing resolution and color quality.^27,110 Nevertheless, individual files of a single whole-slide images can be up to 650 Mb even after compression (20× scan; one focal plane).^20,111 Increasing the magnification or including several z planes can significantly increase file size.⁴

More commonly, organizations currently utilize cloud-based storage solutions to retain digitized slides. This approach is especially suitable for smaller operations, which may lack significant local IT infrastructure and support personnel. Storage, backup, and security are effectively outsourced to a third-party vendor and often can result in better web-based access, faster viewing times, and higher reliability. In contrast, upload times to the cloud may be longer and the cost increases significantly with increasing image file volume.² Especially when working with researchers and/or pathologists outside of the laboratory’s physical location, web-based recall of digital slides facilitates strong collaborative relationships.

Aside from infrastructure and storage space, appropriate database software is needed to manage digital slides, together with unique identifiers and valuable metadata. This enables sensible archiving, searching, and retrieving of information. For a research experiment, it may be useful to record the following for each slide: study type, experiment number, tissue, species, stain, and staining specifics such as antibody catalog number, lot number, and final protein concentration for IHC stains.² With this information, all slides from a specific tissue and/or specific stain or biomarker from a single or multiple study or experiments could be queried. At the very least, however, a unique identifier should be recorded that links the slides to a data record that contains more information, e.g., the patient record, the individual animal number, or an (electronic) laboratory notebook entry. The latter solution, however, does not enable to search the database by metadata features. Depending upon the level of sensitive information stored and the size of the database, the software may also manage and restrict access of individual users to specimens and information.² For example, within the database of an imaging core facility at an academic institution, researchers may only access slides from their own studies, even though the database may include whole-slide images from all researchers and laboratories that utilized the core service. As a database grows and is used by an increasing number of users, standardized terminology, metadata entry, and metadata review/quality control are required to maintain its functionality and searchability. Large operations, such as academic core facilities that provide tissue processing and slide digitization, often have a set-up that links the wet tissue sample, processed tissue block, and stained slide by a barcode that is created and tracked by a laboratory information system. This bar-coding of slides also enables the linking of a digitized slide with the specimen and all its relevant information.²⁷

A detailed description of a WSI database as part of a larger piece on structural reporting is included in the description of the Cancer Electronic Laboratory Management Information and Retrieval developed by the U.S. National Cancer Institute’s Mouse Models of Human Cancer Consortium.¹¹² Although the described format may not fit the specific needs of every research institution or end-user, it provides a solid foundation considering the structure of a novel research database. This database contains all histology slides processed by the Mouse Models of Human Cancer Consortium since 2000. As an application example, entire studies have been conducted and completed by researchers situated at remote locations utilizing this database.

Aside from the simple archiving of slides, the combination of WSI and digital image analysis can aid in recording additional useful annotated information pertinent to each sample. For example, utilizing automated digital pattern recognition image analysis algorithms, proportions of tumor to nontumor tissue for neoplastic samples may assist in selection of appropriate specimens for future studies and can minimize variability inherent in current routine biorepository pathologic evaluation.¹¹³

It is noteworthy that outside of the academic research space (e.g., human diagnostic laboratories, GLP facilities), slide scanning and primary data retention policies are regulated by the FDA. It is suggested to review the FDA’s website for the most up-to-date guidance.

Conclusions

The field of digital microscopy and digital image analysis has opened the door for translational research to not only form world-wide collaborations but also to extract meaningful, robust, and reproducible data from histological sections amenable to rigorous statistical analysis and big data mining. Further developments in the field, such as application of deep learning, hold substantial promise for the expansion of digital tools to enhance the interrogation of tissue specimens. Nevertheless, the pathologist remains the professional who contributes valuable expertise of all study-relevant factors such as influence of preanalytical variables on tissue and staining quality; advanced skills to perform quality assessment of slides, WSI quality, and algorithm performance; and the pathophysiological knowledge to interpret generated data within the context of the research question and limitations of the study design. The success of the research collaboration hinges on early engagement of a pathologist to ensure adequate up-front planning and selection of end-points. Discovery pathologists with training in comparative pathology and translational research remain valuable and crucial members of any research team that is engaged in basic and translational biomedical research generating tissue-derived endpoints.