Artificial Intelligence in Pathology

Abstract

Background

Increasing digitalization enables the use of artificial intelligence (AI) and machine learning in pathology. However, these technologies have only just begun to be implemented, and no randomized prospective trials have yet shown a benefit of AI-based diagnosis. In this review, we present current concepts, illustrate them with examples from representative publications, and discuss the possibilities and limitations of their use.

Methods

This article is based on the results of a search in PubMed for articles published between January 1950 and January 2020 containing the searching terms “artificial intelligence,” “deep learning,” and “digital pathology,” as well as the authors’ own research findings.

Results

Current research on AI in pathology focuses on supporting routine diagnosis and on prognostication, particularly for patients with cancer. Initial data indicate that pathologists can arrive at a diagnosis faster and more accurately with the aid of a computer. In a pilot study on the diagnosis of breast cancer, involving 70 patients, sensitivity for the detection of micrometastases rose from 83.3% (by a pathologist alone) to 91.2% (by a pathologist combined with a computer algorithm). The evidence likewise suggests that AI applied to histomorphological properties of cells during microscopy may enable the inference of certain genetic properties, such as mutations in key genes and deoxyribonucleic acid (DNA) methylation profiles.

Conclusion

Initial proof-of-concept studies for AI in pathology are now available. Randomized, prospective studies are now needed so that these early findings can be confirmed or falsified.

cme plus

This article has been certified by the North Rhine Academy for Continuing Medical Education. Participation in the CME certification program is possible only over the internet: cme.aerzteblatt.de. The deadline for submission is 25 March 2022.

Being concerned with the histomorphological analysis of human tissue specimens, the discipline of pathology plays a key role in the diagnostic workup. For example, guideline-adherent management of numerous oncological conditions requires prior histopathological confirmation of the diagnosis and the number of drugs requiring molecular pathological identification of predictive biomarkers prior to their use is constantly increasing (1, 2). In their daily routine, pathologists generate, analyze and integrate large amounts of data coming from various sources: extensive clinical information, image data from histological and immunohistochemical stainings or molecular pathological data from sequence analyses. The rapid development of whole slide scanning over the last two decades has enabled the digitalization of typically analog microscopic image information in sufficient quality and quantity (3). However, while other imaging disciplines have adopted a largely computer-based work style for years now, digital transformation has only just started in pathology (4). One of the most promising developments in this respect could be the utilization of artificial intelligence (AI) und machine learning (ML) (5). However, AI- and ML-based technologies have not yet been evaluated in a prospective, randomized trial and their adoption in routine pathology is far from being widespread. Thus, the research discussed below should be regarded as proof-of-concept/pilot studies, evaluating various aspects in pathology pertaining to specific AI procedures and identifying potential future areas of application (table 1). While these are representative for the clinical questions addressed, they have not yet been replicated. This review is based on pertinent publications retrieved by a selective search in the PubMed database for the period from January 1950 to January 2020, using the search terms “artificial intelligence“, “deep learning“ and “digital pathology“, as well as the results of our own research.

Table 1. Overview of key studies.

Reference	Entity	Sample size	Method	Type of study	Result
Ehteshami Bejnordi et al. 2017 (14)	Breast cancer	270	Variety of deep learning algorithms	Comparative study of retrospective cases	The best model (AUROC: 0.960) was superior in diagnostic accuracy to eleven pathologists if these had to work under time pressure (AUROC: 0.810).
Mukhopadhyay et al. 2018 (3)	Various	1992	Digitalized tissue slides	Randomized, blinded non-inferiority trial	Digital interpretation is not inferior to microscopy-based interpretation.
Steiner et al. 2018 (15)	Breast cancer	70	Deep convolutional network	Comparative study of retrospective cases	Pathologists achieved higher sensitivity with AI support (increase from 83.3% to 91.2%).
Campanella et al. 2019 (17)	Breast cancerProstate cancer Basal cell carcinoma	15 187	Multiple instance learning	Retrospective study	Without the need for manual tumor annotation, the system achieved AUROCs of up to 0.991 and could independently classify 65–75% of tissue specimens without loss in sensitivity.

AUROC: area under the receiver operator characteristic curve

Artificial intelligence and digital image analysis

AI describes the process of teaching a machine to solve a problem without the need to explicitly specify each step to the solution. AI research and development is a subfield of computer science. Machine learning, in turn, is concerned with specific methods that can be used to implement AI. Here, a distinction is sometimes made between classical machine learning and newer approaches, such as deep learning. A number of developments in the last decade have led to a steep increase in AI applications: improved algorithms, improved hardware and a rapidly growing amount of obtainable data. These developments paralleled the trend in pathology towards increasing digitalization (figure 1).

Figure 1.

Key developments in artificial intelligence and pathology (27– 31).

DICOM, digital imaging and communications in medicine; FDA, Food and Drug Administration; FISH, fluorescence in situ hybridization;

PACS, picture archiving and communication system

These key developments have occurred mostly in the field of computer-based, automated processing of image data, which is also referred to as machine vision or computer vision. Such solutions primarily rely on the application of artificial neural networks. The basic idea underlying these complex algorithms is to mathematically replicate biological neural systems. Frequently, these models are trained using millions of images of well-characterized datasets, such as the ImageNet dataset. In the transfer-learning step, such pre-trained networks are then applied with some modifications to other questions (table 2).

Table 2. Explanation of key terms from the field of artificial intelligence.

Term	Description
Artificial intelligence	Artificial intelligence describes the ability of a computer system to replicate intelligent/human behavior. Machine learning methods are used for this purpose.
Machine learning	Machine learning describes computer algorithms that improve with experience and generate artificial knowledge. This allows problems to be solved without the need to explicitly specify (i.e. program, for example) each individual solution step.
Deep learning	Deep learning is a subfield of machine learning which is in particular concerned with the use of artificial neural networks.
Artificial neural network	An artificial neural network is a computer model mimicking biological neural networks. It consists of artificial neurons that can be arranged, for example, in layers/planes, one after the other. They are often used in deep learning applications.
Convolutional network	A convolutional network is a subtype of artificial neural networks which reduces the input of presumably relevant features, using specific folding matrices. Convolution networks are primarily used in automated image recognition applications.
Recurrent neural network	In recurrent neural networks, weighted connections exist not only in the direction of the output layer, but also in preceding layers/levels. This can create a kind of memory which is useful, for example, in the recognition of sequences.
Support vector machine	Support vector machine describes a mathematical method of pattern recognition and is commonly cited as an example of classical machine learning. Classification of objects is achieved by assigning objects to classes in such a way that the area between the individual classes becomes as large as possible.
Transfer learning	Transfer learning is used, for example, in artificial neural networks and describes the process by which a network is first trained on certain data and the pre-trained network is then applied to other data.
Graphical processing unit	Hardware component commonly used for training artificial neural networks.
Multiple instance learning	With this method, not every single training example is annotated; instead, so-called instances, for example all image tiles, are assigned to a slide. This simplifies the annotation process.

Recently, AI-assisted approaches have increasingly been evaluated in medical settings and could find more widespread use, especially in imaging disciplines, in the near future. In radiology, for example, a number of companies are already offering products designed to address specific routine diagnostic questions; however, none of these products are being used on a nationwide basis (5).In pathology, this process happens even more hesitantly, because to date whole slide scanners are rarely used for routine diagnosis in Germany. The possible reasons for this are complex, ranging from high costs of investment to security questions to reservations among pathologists. However, some commercial and academic institutions are undertaking pioneering work in this area. In the institutes of pathology in Leeds, Utrecht, Pittsburgh, and New York, digital interpretation has already been partially or predominantly implemented and some companies are already offering specific CE-certified pathology products in certain market segments (5, 6). Besides whole slide scanning, digital reporting comprises automated barcode-based collection of case numbers, speech recognition-assisted dictation of findings and automated transmission of findings to the hospital information system (HIS), among others. A blinded randomized noninferiority study of 1992 cases showed that computer-based digital interpretation was on a par with microscopy-based analog diagnosis (3). In addition, Ho et al. predicted potential cost savings of US$ 12.4 million for a university center over a period of five years, based on model calculations. Of these, US$ 5.4 million are attributable to improved accuracy of diagnoses and resulting reductions in treatment costs (7). Moreover, Stathonikos et al. reported shorter processing times, especially in complex cases (6). However, this exclusively applies to a digital-based approach; for AI applications in pathology, comparable surveys have not yet become available. Both AI support of routine diagnosis in classical pathology and the introduction of novel computer-assisted diagnostic methods are conceivable (figure 2).

Figure 2.

Example of a deep learning model, designed to differentiate colorectal cancer from normal colorectal mucosa. On the left, the conventional histological input image; on the right, highlighting of the tissue according to the result of classification by the artificial intelligence model .First, individual image sections (tiles) are classified by the artificial neural network and then each individual tile is color-coded based on prediction probability: higher probability of the class “tumor“: red; higher probability of the class “normal mucosa”: green (unpublished data, Försch et al.).

Artificial intelligence to support routine pathological diagnosis

AI and ML algorithms are especially useful for addressing recurrent questions with limited complexity. In routine pathological diagnosis, this includes, among others, analyzes of tissue specimens from large screening programs, such as colorectal cancer screening, and cases where many similar slides are submitted, for example sections of prostate specimens. Shifting time-consuming repetitive screening tasks to AI algorithms would allow pathologists to dedicate more time to more sophisticated activities, such as interpreting predictive or prognostic biomarkers in the context of individual clinical findings. Research based on manual extraction of image properties and classical morphometry dates back to the 1980s (8– 10). At the same time, work to advance the statistical methods was undertaken and first applications for these refined techniques were identified in pathology. In a retrospective analysis, it was possible to differentiate between breast cancer and healthy breast parenchyma with 98.8% accuracy in several benchmark datasets, using feature extraction and a support vector machine (SVM) model-based classifier (11). Similar studies were conducted for prostate cancer and oral cavity squamous cell carcinoma (12, 13). These achieved accuracy levels of 70.8% and 99.1%, respectively. With the help of deep learning and artificial neural networks, classification accuracy could be further improved. In another retrospective study, Ehteshami Bejnordi et al. focused on the detection of lymph node metastases in patients with breast cancer. They compared different models submitted by participants of an annual programming competition in biomedical imaging.The top-performing algorithm, a convolution network based on Google technology, achieved an area under the curve (AUC) value in the receiver operating characteristic curve (AUROC) of 0.994, corresponding to an average false positive rate of 1.25. In the receiver operating characteristic curve, sensitivity is usually plotted against 1-specificity; the integral (AUROC) serves as a quality criterion for predictive power. In addition, the authors compared the best models with the performance of the pathologists. In this comparison, the expert pathologists achieved, in the absence of time constraints, marginally better accuracy values (AUC 0.966) compared to the algorithms (AUC 0.960). However, these values significantly declined as soon as the interpreting pathologist was under time pressure (AUC 0.810) (14). In another study addressing the same question, diagnostic accuracy was assessed with and without AI-based support. In cases with difficult to identify micrometastases, sensitivity improved from 83.3% (human alone) to 91.2% (human in combination with machine) (p = 0.023). In addition, the authors reported higher overall accuracy for the human + machine team compared to interpretation accuracy achieved by human alone or machine alone (15).

The fact that so far only very few pathology laboratories have established a fully digital workflow is a key barrier to the implementation of such models in routine clinical diagnosis. Hybrid methods, including integration of augmented/virtual reality (AR/VR) could be important intermediate steps. Using augmented reality microscopy, where the microscopic image is obtained by a camera and presented to a pre-trained neural network, areas suspicious of containing tumor were marked in the pathologist’s field of view, using a light pointer (16). In Germany, similar approaches are being pursued, for example, in the development of clinical decision support systems (CDSS).

The strategies described to date are primarily based on the supervised learning method. This implies that each image presented to the network during training must first be annotated by a human expert. Since efficient training often requires thousands of example images, this results in a considerable expenditure of work and time. At the same time, expertise in pathology is rare and cannot be easily obtained from other sources, such as crowdsourcing. The method of semi-supervised or unsupervised learning could be a solution to this problem. Multiple instance learning is a method where first all image sections are classified and then only the sections with the lowest classification error are used for training. An annotation of specific image areas is not required; only the entire slide is labeled, e.g. tumor yes/no. Using this method and altogether 44 732 slides of 15 187 patients, Campanella et al. achieved AUROC values of up to 0.991 in the diagnosis of prostate cancer, basal cell carcinoma of the skin and lymph node metastases in a retrospective setting. The discussed implications for routine clinical diagnosis are of interest: If such a system would be used for 65% to 75% of the histopathological specimens received, sensitivity would still be at almost 100% (17).

New diagnostic capabilities through artificial intelligence

Artificial intelligence-assisted image analysis for prognosis prediction

In pathology, important diagnostic queries include estimation of a patient’s prognosis and prediction of potential response to treatment. Frequently, additional immunohistochemical or molecular biological testing, such as next generation sequencing, is used to determine prognostic and predictive biomarkers. Here, again, the application of artificial intelligence in combination with digital pathology would lead to a paradigm shift. AI algorithms can identify subvisual structural characteristics which the human eye is not capable of quantifying, thereby establishing a new class of morphology-based biomarkers with prognostic or predictive validity. In a retrospective study, for example, the prognosis of breast cancer patients was successfully estimated based on hematoxylin-eosin (H&E)–stained slides, using feature extraction and machine learning (18). Image properties were extracted automatically and thousands of properties were identified on cell level and tissue level. The log-rank test found a highly significant association between the prediction of the model and the survival of the patients (p ≤ 0.001), independent of other clinical or pathological risk factors. Patients classified as high-risk patients by the model had a 5-year survival rate of 68.7%. By contrast, 84.5% of the low-risk patients were still alive after five years. In addition, the histopathological criteria contributing most to the respective classification were identified by statistical analyses. A similar method, which was based on deep learning, was described for colorectal cancer (19) and other conditions. First, a deep convolution network was used for image processing. Its output then served as the input of a second network, a so-called recurrent neuronal network with long short-term memory function. Again in retrospective proof-of-concept studies, this method was able to detect microsatellite instability in gastrointestinal tumors and identify the molecular subtype of bladder cancer (20, 21).

Artificial intelligence-based analysis of genetic data

The concept of genotype-phenotype coupling implicates that it is highly likely that in tumor tissue—for example, with a mutation in a key gene—changes in morphology are also found. Besides the ability to predict molecular or clinical parameters based on histomorphological data, AI applications will play an increasingly important role in the analysis of molecular pathological data. One example is the classification of deoxyribonucleic acid (DNA) methylation profiles of squamous cell carcinomas of the lung using deep learning to distinguish a metastasis of a head and neck carcinoma from primary lung cancer (22). Likewise, AI methods play an increasingly important role in multi-omics-based molecular tumor classifications (23). Even though no immediate clinical use has yet been established, evidence of the future complementary significance of complex molecular and morphological profiles is already emerging (24).

Integration of histomorphological, molecular pathological and oncological data

The AI approaches mentioned above allow to predict specific molecular or clinical properties based on histomorphological images or to establish pathological classifications from molecular data. A promising approach to arrive at an even more accurate prognosis or prediction in tumor disease could be to use a combination of various data modalities as input to an AI model. Here, a pathology-specific concept is conceivable that, for example, would integrate histological with immunohistochemical and molecular data. An intriguing example of this was provided by Mobadersany et al. who, in a retrospective study of patients with glioma, included genetic information in addition to image information (25). The combination of image and mutation data achieved better prognostic prediction probability compared to the use of image data alone or mutation data alone (p = 0.0106), expressed as an increase in C index by 5% from 0.754 to 0.801. In the personalized medicine of the future, a multimodal biomarker analysis with integration of morphological, radiographic, laboratory and clinical parameters with genomic and proteomic data could present an enormous challenge due to the complexity of the information that could only be overcome by using AI approaches. However, this would require substantial IT and clinical expertise as well as highest data quality; consequently, the availability of such a solution for routine clinical applications is still a long way off.

Challenges in the implementation of artificial intelligence-based diagnosis

Although artificial intelligence and machine learning have the potential to revolutionize the specialty of pathology, there are a number of significant challenges to their translational implementation. There is a strong positive correlation between the accuracy of an AI algorithm and the amount of data used.In the study by Campanella et al., the validation error decreased approximately by a factor of 10 when 100 times more cases were analyzed (17). However, only a fraction of the histopathological specimens are available in a digital format allowing for computerized analysis. Furthermore, digitalization always requires substantial initial investments. Even though the proportion of digitalized information will significantly increase in the medium term, i.e. in the next decade at the latest, a detailed review and exact description of these data by expert pathologists is lacking. This has major implications for the quality of the data—another factor with significant impact on the accuracy of an AI model (garbage in, garbage out problem). The guide “Digital Pathology“ published by the Professional Association of German Pathologists (Berufsverband Deutscher Pathologen e. V.) offers assistance with the digitalization of pathology. It explicitly encourages the use of digital methods and focusses on the freedom of choice of the method and the pathologist’s responsibility for the selected path to diagnosis (26). At present, the greatest obstacle to the use of AI-based techniques is the almost complete lack of prospective, randomized, multicenter trials evaluating the benefit for pathologists on the one hand and patients on the other hand. Such studies are urgently needed to identify AI solutions that really improve patient-related outcomes. Besides image data, high-quality clinical data are essential for this kind of research. This primarily applies to a potential predictive application of AI-based methods. If these concerns are addressed, the use of AI and digital pathology has the potential to transform the specialty and help pathologists to do their work faster and with greater accuracy.

Questions on the article in issue 12/2021:

Artificial Intelligence in Pathology

The submission deadline is 25 March 2022. Only one answer is possible per question. Please select the answer that is most appropriate.

Question 1

When were the first whole slide scanners used in studies?

1. From the early 1980s

2. Mid 1970s

3. From the early 1990s

4. Around the year 2000

5. From the year 2010

Question 2

Which of the artificial intelligence methods described in the article does not require annotation, because a specific label is assigned to the entire slide (e.g. tumor yes/no)?

1. Single strategy learning

2. Multiple instance learning

3. Picture-based learning

4. Simple strategy learning

5. Iterative learning

Question 3

The Turing Test (introduced by Alan Turing) laid an important foundation for the concept of artificial intelligence.

When did Alan Turin first propose this test?

1. In 1945

2. In 1970

3. In 1965

4. In 1950

5. In 1980

Question 4

In a retrospective study by Campanella et al., very high AUROC values were achieved (up to 0.991). Which tissues, among others, were evaluated in the study?

1. Prostate cancer and basal cell carcinoma

2. Basal cell carcinoma and renal cell carcinoma

3. Rectal cancer and neuroblastoma

4. Ovarian cancer and renal cell carcinoma

5. Neuroblastoma and breast cancer

Question 5

The accuracy of an AI algorithm correlates with the amount of data used. By approximately what factor did the validation error decrease in the study by Campanella et al. when 100 times more cases were used?

1. Approx. by a factor of 2

2. Approx. by a factor of 10

3. Approx. by a factor of 20

4. Approx. by a factor of 50

5. Approx. by a factor of 100

Question 6

What is usually plotted in the receiver operating characteristic curve?

1. Specificity plotted against 1-sensitivity

2. Specificity plotted against 1-specificity

3. Sensitivity plotted against specificity

4. Sensitivity plotted against 1-specificity

5. 1-specificity plotted against 1

Question 7

In a retrospective study, the prognosis of breast cancer patients was successfully estimated based on hematoxylin-eosin (H&E)–stained slides, using feature extraction and machine learning. What was the difference in 5-year survival rates between high-risk and low-risk patients?

1. 50.2% (high risk) vs. 70.5% (low risk)

2. 68.7% (high risk) vs. 84.5% (low risk)

3. 32.1% (high risk) vs. 60.4% (low risk)

4. 46.3% (high risk) vs. 56.4% (low risk)

5. 25.7% (high risk) vs. 49.6% (low risk)

Question 8

In a retrospective comparative study of 270 cases of breast cancer, various deep learning algorithms were tested in comparison with eleven expert pathologists. In this study, under what conditions was a higher accuracy achieved by AI than by expert pathologists?

1. If the slides were very large.

2. If multiple slides were available for each patient.

3. If AI received additional information on the patients (age, pre-existing conditions).

4. If the expert pathologists were under time pressure.

5. If the quality of the slides was low.

Question 9

In a study by Steiner et al. on the diagnostic accuracy achieved with and without AI assistance, the sensitivity for the diagnosis of difficult to identify micrometastases in lymph nodes was assessed. What result is reported by the authors with regard to the diagnostic approach?

1. The combination of human and AI achieved a higher sensitivity than human alone (91.2% vs. 83.3%).

2. Human alone achieved the highest sensitivity (91.2%).

3. Machine alone achieved a higher sensitivity than human alone (99.6% vs. 76.4%).

4. The combination of human and AI achieved a similar sensitivity than human alone (83.7% vs. 83.3%).

5. The overall accuracy of interpretation of the human–machine team was lower compared to the interpretation by human alone and machine alone.

Question 10

What is the name of the prototype of a microscope in which the microscopic image is captured by a camera and presented to a pre-trained artificial neural network so that tumor-suspicious areas can be marked for the pathologist using a light pointer?

1. Digital learning microscope

2. Digital intelligent microscope

3. Virtual intelligence microscope

4. Augmented reality microscope

5. Augmented knowledge microscope