Machine learning classification of placental villous infarction, perivillous fibrin deposition, and intervillous thrombus

Abstract

Introduction:

Placental parenchymal lesions are commonly encountered and carry significant clinical associations. However, they are frequently missed or misclassified by general practice pathologists. Interpretation of pathology slides has emerged as one of the most successful applications of machine learning (ML) in medicine with applications ranging from cancer detection and prognostication to transplant medicine. The goal of this study was to use a whole-slide learning model to identify and classify placental parenchymal lesions including villous infarctions, intervillous thrombi (IVT), and perivillous fibrin deposition (PVFD).

Methods:

We generated whole slide images from placental discs examined at our institution with infarct, IVT, PVFD, or no macroscopic lesion. Slides were analyzed as a set of overlapping patches. We extracted feature vectors from each patch using a pretrained convolutional neural network (EfficientNetV2L). We trained a model to assign attention to each vector and used the attentions as weights to produce a pooled feature vector. The pooled vector was classified as normal or 1 of 3 lesions using a fully connected network. Patch attention was plotted to highlight informative areas of the slide.

Results:

Overall balanced accuracy in a test set of held-out slides was 0.86 with receiver-operator characteristic areas under the curve of 0.917 to 0.993. Cases of PVFD were frequently miscalled as normal or infarcts, the latter possibly due to the perivillous fibrin found at the periphery of infarctions. We used attention maps to further understand some errors, including one most likely due to poor tissue fixation and processing.

Discussion:

We used a whole-slide learning paradigm to train models to recognize three of the most common placental parenchymal lesions. We used attention maps to gain insight into model function, which differed from intuitive explanations.

INTRODUCTION

The placenta

The placenta is the temporary organ formed during gestation that performs nutritional, respiratory, excretory, endocrine, and immune-regulatory functions throughout fetal development.[1–3] The normal placental disc is comprised primarily of terminal villi, which contain fetal capillaries and act as the site of uptake of oxygen and nutrients and excretion of carbon dioxide and waste.[4]

Placental parenchymal lesions

Vascular and/or inflammatory pathologic conditions can affect placental function and compromise pregnancy, and identification of these lesions histologically can be important in assessment of placental function. Several lesions have been identified that form in the villous portion of the placenta. We have chosen to focuses on a subset of macroscopic lesions that cause localized injury or replacement of villi.[5,6] Specifically, we focus on villous infarctions, intervillous thrombi, and perivillous fibrin deposition.

Villous infarctions, (hereafter infarcts), are areas where there is a loss of maternal blood flow with subsequent ischemic necrosis of the villi. They are thus analogous to ischemic infarcts seen at other sites in the body including the myocardium, kidney, and central nervous system. Infarcts are seen in ~8% of placentas examined at our institution. They are associated with hypertension in pregnancy, preeclampsia, growth restriction, stillbirth, and risk of cerebral palsy.[7–11] Microscopically, infarctions are characterized by loss of definition in the villi and collapse of the intervillous space or replacement by loose blue-grey necrotic material (Figure 1).

Figure 1. Placental anatomy and parenchymal lesions:

Normal placental histology showing villi are bathed in maternal blood (A). Infarctions are due to loss of maternal blood flow and characterized by indistinct villi that are either back-to-back or with loose material between (B). PVFD shows loss of syncytiotrophoblast layer from the villous surface with replacement of the intervillous space by dense pink fibrin (C). IVT displace villi and show characteristic lines of Zahn (D). Scale bar 200 μm.

Intervillous thrombi (IVT), are foci of clotted blood in the intervillous space. In contrast to other macroscopic lesions, IVT show no clear association with adverse pregnancy or neonatal outcomes.[12–14] However, they are frequently encountered it is of value to distinguish them when there is high interobserver variability or in low resource settings. Microscopic examination of IVT show areas with few to no villi containing alternating areas of fibrin and red cells – the Lines of Zahn (Figure 1).

Perivillous fibrin deposition (PVFD) can present in several forms, of which two are considered here. Massive perivillous fibrin deposition (MPVFD) involves at least 30% of the placental parenchyma with significant compromise of placental function and associated fetal growth restriction and stillbirth.[15–17] Conversely, focal PVFD forms foci up to a few cm in greatest dimension involving a small proportion of the placenta. Pathogenesis of focal PVFD is unclear but they may represent a reparative response following trophoblast injury due to turbulent flow.[18,19] Focal PVFD may also be seen as a feature of chronic inflammatory pathology.[20] Despite these striking clinical differences, MPVFD and focal PVFD appear similar microscopically, and may represent ends of a spectrum with “borderline” MPVFD between.[16] The intervillous space is filled with dense fibrin, while terminal villi lose their outer syncytiotrophoblast layer (Figure 1).

While these lesions are clinically significant, there is high interobserver variability in their diagnosis, limiting their utility in broader clinical practice. Literature is sparse, but in examples given to community pathologists, misdiagnosis rates of 68.4% for MPVFD, 57.1% for IVT, and 32.4% for infarction were reported, with most of the errors being underdiagnosis.[21] This underlines the sparseness of perinatal pathology expertise and the potential utility of diagnostic aids.

Machine learning (ML) and digital pathology

Interpretation of pathology slides has emerged as one of the most successful applications of AI/ML in medicine with applications ranging from cancer detection and prognostication to transplant medicine.[22–26] In the realm of placental pathology, accurate diagnosis of macroscopic lesions requires expert microscopic examination.[18,27] Machine learning approaches promise to reduce interobserver variation and make expert-level examination available in general practice and in low and middle income countries.[28–30]

Digital pathology challenges machine learning practitioners due to the large size of scanned whole slide images (WSI). While typical image classification software uses small images as input (e.g. 256 × 256 pixels), WSI are routinely 120,000 × 80,000 pixels.[31,32] In this task, models and humans are confronted with a similar problem – identifying which parts of the slide diagnostically relevant. To address this, we employ multiple instance learning, a domain of machine learning where models are given several related datapoints and asked to infer a single label.[33] MIL has demonstrated remarkable success in identifying cancers of unknown primary, and cardiac allograft rejection and is a component of some clinically used systems.[22,34–38]

From slide to diagnosis, whole slide learning in practice

A notional illustration of whole-slide classification is given in Figure 2. A hematoxylin and eosin (H&E) stained slide is examined. The slide contains an area of infarct (red outlined squares) with the remainder normal placental tissue (black squares). The tissue is divided into patches – smaller 256 × 256 pixel images that are typical of modern ML. The patches are fed to a fixed feature extraction subnetwork, EfficientNetV2L.[39] This subnetwork has been pre-trained to parse images into common patterns like edges, simple shapes, and textures.[39,40] This results in a feature vector, a list of numbers indicating how much the image represents each pattern. To identify salient patches, the feature vectors are fed into an attention subnetwork. In a properly functioning model, feature vectors from the infarct (red-shaded) are assigned a high attention while those of normal tissue (grayscale) are assigned a low value. An average feature vector, weighted by the attention values, is generated (weighted features). In a well-functioning model this will largely reflect the lesion. The weighted feature vector is fed into a classifier subnetwork that generates a value between 0 and 1, with values >0.5 classified having infarction. During training, the parameters of the attention subnetwork and classification subnetwork are tuned to maximize accuracy in a set of training slides. The feature extraction subnetwork is fixed. By tapping into the attention subnetwork, the attention value assigned to each feature vector and therefore each patch can be visualized as an attention map, with high attention (yellow) given to the diagnostic areas and low attention (blue) to normal placental tissue. For this work, four attention subnetworks and four classifications are generated, one each for normal, infarct, IVT, and PVFD.

Figure 2. Whole-slide learning method - diagram:

A whole slide image (WSI) is divided into a set of smaller images (patches). Images pass through a feature extraction network to become feature vectors. The attention subnetwork generates an attention value for each vector and uses that value to produce a weighted average (weighted features). The classifier subnetwork then generates a single label for the whole slide. Attention values for each patch are plotted on the attention map (blue-yellow: low-high attention). Note that a separate attention subnetwork and classifier subnetwork exist for each diagnosis, though only that for infarction is shown.

The goal of this study is to use whole slide learning to classify infarcts, IVT, and PVFD. We have selected these lesions because they are commonly encountered in pathology practice. Additionally, diagnosis of these lesions involves integrating information across large regions of the slide drawing on the strength of the whole slide learning method. Finally, to the best of our knowledge, models for classifying these lesions have not been previously reported.

METHODS

Patients, diagnoses

We identified patients that underwent delivery and placental examination at our institution between 2011 and 2022 and had slides available for scanning. Our scanning program is ongoing and the material scanned represents a mixture of prospectively scanned slides and targeted archival retrieval, e.g. for MPVFD cases. Slides were digitized on a Leica GT450 scanner with a 40x objective magnification (0.263 microns per pixel). Corresponding placental pathology reports were extracted from the institutional electronic data warehouse (EDW). Natural language processing was used to parse reports.[41] Placentas with retroplacental hemorrhage / hematoma or chorangioma or with multiple classes of lesions were excluded. Placentas were classified as having infarct, IVT, PVFD, or none of these (normal). PVFD cases included MPVFD and focal PVFD, which was defined as a macroscopic placental lesion characterized by encasement of villi in perivillous fibrin. Both lesions are distinguished from the normal degree of perivillous fibrin by the formation of macroscopic lesions.[42]

Patient and diagnosis data was stored in REDcap.[43] Slide images were reviewed by a perinatal pathologist aware of the original diagnosis using Digital Slide Archive (DSA).[44] The study was approved by our local institutional review board (STU00214052).

Feature extraction

One slide was used per patient. Non-tissue regions were identified and ignored by using Otsu’s method on low-power slide images. Machine learning was performed using TensorFlow version 2.8. Portions of the 10x image containing tissue were broken into 256 × 256-pixel patches with 64-pixel overlaps. This resulted in 5,000 – 16,000 patches per slide, depending on the amount of tissue present. Patches were submitted to the feature extraction backbone of EfficientNetV2L, trained on ImageNet.[39]

Attention, classification

Feature vectors were submitted to the network as N × D arrays, where N is the number of patches analyzed from that slides and D is the number of features in the vector from one patch, 1536 for EfficientNetV2. Feature vectors underwent further dimensionality reduction using a fully connected layer with 512 outputs and a rectified linear unit (relu) activation. Attention is generated using the dot product of two parallel 256-neuron layers with hyperbolic tangent and sigmoid activations. Four attentions are generated in parallel, one for each potential diagnosis. Attention values are used in the weights for weighted pooling of the dimensionality reduced layers, which then go through four fully connected classifier layers to produce one score for each diagnosis. The four scores undergo a softmax transformation to produce the final output – four scores that sum to 1.0 and represent the relative confidence of the model that the slide represents each diagnosis. For example, scores of [1, 0, 0, 0] would represent complete confidence that the slide is normal placenta, and not infarct, IVT, or PVFD (respectively). Scores of [0, 0.5, 0, 0.5] represent a slide that the model is equally confident could be infarction or PVFD. Note that confidence is a reflection of model output, not epistemic uncertainty or biologic reality. The model can by highly confident and dead wrong.

Training, validation, testing

The dataset consists of 790 cases, including 219 normal, 193 infarct, 186 IVT, and 92 PVFD. The dataset was split 70:15:15 into training, validation, and testing sets. The splits were stratified, such that the ratio of each diagnosis was consistent between the sets. The model was trained to minimize the categorical crossentropy between the actual and model-predicted diagnoses. The Adam optimizer, an initial learning rate of 1e-4, cosine decay with 1000 steps and restarts were used. Model performance was monitored throughout training by following the categorical crossentropy of the validation data. Up to 100 epochs were allowed, but training was stopped when the categorical crossentropy of the validation data did not improve after 15 epochs.

Attention maps

Attention values were output from the model as 1 × N arrays, with 1 attention value for each of the N examined patches. Attention values were normalized with 255 indicating the highest attention. Patches not analyzed were assigned a value of 0. TFR files contain similar 1 × N arrays of each tile’s left edge and upper edge, which are used to produce a 2 dimensional array, where each pixel corresponds to a single patch. False color and overlay of slide images was performed using MatPlotLib.

Convolutional whole slide models

We tested two alternative models for whole slide learning (Figure 3). These models use the same feature extractor network to generate feature vectors but use convolutional neural networks in subsequent steps. Convolutional neural networks have shown extreme facility in classifying images that present as 3-dimensional matrices with the shape W × H × C, where W and H are the width and height of the image and C is the number channels, often 3 (red, green blue).[45] While the main model treats each feature vector independently, the convolutional models arrange the feature vectors based on their location in the original image. The resulting data structure forms a 3-dimensional matrix with the shape of W × H × F, where W and H are the number of patches horizontally (Width) and vertically (Height) present in the image and F is the number of features in each patch. Convolutional neural networks can treat this matrix as they would an image, identifying patterns among neighboring patches. In principle, these models should be superior at tasks that require identification of patterns larger than a single patch.

Figure 3, Convolutional-based models:

A whole slide image (WSI, detail of slide from Figure 2) is divided into a set of smaller images (patches). Images pass through a feature extraction network to become feature vectors. Unlike the dense attention model, the values across the slide from a single feature formed into 2-dimensional feature maps. There is one feature map for each of the 1280 features. A simplified 2×2 kernel is shown, which outputs the average of the values in the upper left and lower left squares (tinted aqua). The kernel is moved across and down the feature map, giving the Convolutional output shown. In the convolutional simple model, the maximum value for each feature map is used to generate a feature vector, which is then classified (upper path). For the convolutional attention model, an additional convolutional step is performed (not shown) and the feature vectors are submitted to an attention subnetwork.

We used two convolutional models, “convolutional simple” and “convolutional attention”. In the convolutional simple model, the structured matrix of feature vectors is submitted to a 2-dimensional convolutional layer with 128 filters and a kernel size of 3. Kernel size of 3 means that the feature vectors are studied in 3 × 3 groups – that is, each vector is studied in the context of its immediate neighbors. Each filter is a different pattern 3 × 3 pattern that is sought in each 3 × 3 region for each feature. For example, a filter might check whether the 2^nd feature in the feature vector is above a certain threshold in the central patch, the patch up and to the left, and the patch down and to the right, forming a diagonal line. After convolution, the simple model uses 2-dimensional global max pooling to find the 3 × 3 region with the highest signal for each filter. That information is submitted to a fully connected classifier layer that outputs softmax values for each of the 4 classes.

In the convolutional attention model, the structured matrix of feature vectors is submitted first to a convolutional layer with 256 filters and a kernel size of 3, followed by a second convolutional layer with 128 filters and a kernel size of 3. The first convolutional layer is similar to the convolutional layer in the convolutional simple model, albeit with more filters. However, the second convolutional layer operates on the output from the first convolutional layer. This allows more complex patterns to be built up, for example an open diamond adjacent to a diagonal line. At this point, the output is split into 4 branches – one for each class. For each class, an attention map is generated by using a convolutional layer with a single filter and a kernel size of 1, that is, it uses only the information from a single patch at a time. A consensus feature vector is generated by calculating an average of the 2^nd convolutional layer output, weighted for the attention. Each of the 4 consensus feature vectors is assigned a score, then they are brought together and go through a softmax to produce the expected output of 4 values, 1 for each class, that sum to 1.

RESULTS

We developed at dataset of 790 cases, including 219 normal, 193 infarct, 186 IVT, and 92 PVFD. Slides were split into a training set of 543 cases, 120 validation cases, and 127 test cases. The model was trained through 29 epochs before early stopping was triggered due to non-improvement in the validation categorical crossentropy (Supplementary Figure 1). The test set was evaluated, producing a balanced accuracy of 0.86 and a Matthews correlation coefficient (MCC) of 0.86. Further metrics are presented in Table 1 and Figure 4.

Table 1:

Performance of the dense attention model

	Precision	Recall	F1-score
Normal	0.94	0.96	0.95
Infarct	0.93	0.87	0.9
IVT	0.91	0.94	0.92
PVFD	0.67	0.67	0.67
Overall	0.9	0.9	0.90

Figure 4, Confusion matrix for the dense attention model:

Values show the proportion of cases classified into each class for each lesion (or normal).

Model performance

We compared the performance of our network with two convolutional neural networks to capture information about patches in the context of their neighbors (see Methods). The convolutional information is then either simply averaged (“convolutional simple”) or used to generate an attention map with weights assigned to high attention groups of patches (“convolutional attention”). We generated receiver-operator characteristic curves for each classifier for each lesion (Supplementary Tables 1, 2, and 3, Supplementary Figures 2 and 3). Performance was slightly inferior, with balanced accuracies 0.79 and 0.82 and MCC of 0.78 and 0.8 for the convolutional simple and convolutional attention models, respectively.

Examination of errors

Some trends emerge in the errors. We had the largest number of normal slides, and they were identified with the highest accuracy. Concordantly, we had the smallest number of slides with PVFD and they showed the lowest sensitivity. Normal slides were rarely called as one of the lesions, while each lesions was sometimes miscalled as normal. For each class, we manually reviewed cases in the test set that were misclassified, and discuss a few illustrative examples.

One case of subchorionic PVFD was miscalled as normal, possibly due to large portions or normal. Small amounts of subchorionic fibrin are frequently encountered, and are included in the training set as “normal” if they are below the diagnostic threshold (Figure 5A). Some cases of normal were miscalled as IVT (Figure 5B). One IVT was surrounded by compressed villi, and was misclassified as an infarct (Figure 5C).

Figure 5, Morphology of misclassified cases:

A: Focus of PVFD miscalled as normal, possibly due to large portions or normal. B: Section of normal placenta miscalled as IVT. The basis for this error is non-obvious. C: IVT miscalled as infarct, possibly related to compressed villi around the edge. Scale bar = 1 mm

DISCUSSION

Placental parenchymal lesions are commonly encountered and carry significant clinical associations. [46,47] However, they are frequently missed or misclassified by pathologists in general.[21] Artificial intelligence raises the possibility of rapid, uniform diagnosis of these findings. Our findings indicate that lesions can be distinguished from one another and from normal tissue.

Our study is noteworthy for the use of whole-slide learning. In contrast to approaches in which human annotators select a portion of the image for analysis, the model ingests the entire image to produce a unified diagnosis. This represents an increase in sophistication for the model, but also exposes challenges and errors that stem from the heterogeneity of data across a whole slide.

Even the slides of normal placenta in our data set contain multiple tissue types. In addition to the stem and terminal villi, sections of normal parenchyma often contain decidua – uterine lining transformed under the influence of progesterone, and chorionic plate – a layer of fibrous tissue supporting the largest fetal vessels and dividing the villi from the amniotic fluid. Pathology guidelines recommend sampling the interface between normal and abnormal tissues.[5] Therefore slides of parenchymal lesions generally contain stem and terminal villi as well as the lesion of interest. Lesions themselves are often heterogeneous, with both IVT and infarcts showing villi trapped in PVFD around their edges.[48]

Attention maps in explaining and improving models

When human experts and AI disagree, methods are needed to evaluate why the AI renders its diagnosis. However, AI models often operate as “black boxes” that lack the ability to explain their predictions. This hampers efforts to improve models in development and reason with their findings after deployment. [49] While several approaches have been employed, this study uses attention maps.[50–54]

We examined the attention maps for appropriately and misclassified cases to gain insight into model performance and possible improvements. In an appropriately classified case of IVT, the model showed high attention for IVT within the IVT (Figure 6). IVT frequently have a thin surrounding layer of fibrin with or without entrapped villi. Examination of the attention values for PVFD showed the model appropriately attending to areas of fibrin. This did not result in misclassification.

Figure 6, attention maps for understanding appropriate classification.

A: Low power view showing the interface between IVT (T) and normal villi (V). B: Attention for IVT is appropriately highest within the IVT and overlapping lines of Zahn. C: Attention for PVFD highlighting expected thin rim of fibrin between IVT and normal tissue. Scale bar: 250 μm. Attention: Blue – low, Yellow – high.

To examine the utility of attention maps for understanding misclassification, we examined the attention values for an IVT that was misclassified as infarct (Figure 7). In contrast to the more typical IVT seen in Figure 5, the perivillous fibrin surrounding this IVT contains large numbers of compressed villi. Despite significant attention in lines of Zahn and perivillous fibrin, a single high-attention patch seems to have overthrown the diagnosis.

Figure 7, attention maps for understanding inappropriate classification.

A: Low power view showing the interface between thrombus (IVT) and rim of compressed villi (V). B: Attention for IVT is appropriately highest within the thrombus and overlapping lines of Zahn. C: Attention for infarct highlighting expected a single very high-attention focus. D: Expected rim of PVFD between thrombus and normal tissue. Scale bar: 250 um. Attention: Blue – low, Yellow – high.

Limitations and future work

As a first attempt, we have chosen to focus on a subset of the most common lesions. Other parenchymal lesions include chorangiomas, retroplacental hematomas, subchorionic cysts, septal cysts, choriocarcinomas, macroscopic foci of avascular villi, and abscesses.[5,55–59] Additionally lesions may be seen together either coincidentally or in hybrid lesions, such as the infarction hematoma (infarct + retroplacental hematoma).[60,61] Future studies will expand the number and diversity of lesions studied.

Differences in staining over time and between institutions are a major barrier to generalizability of ML models.[62,63] Innovative approaches, including color normalization, color augmentation, and style transfer have been used to address this challenge.[63,64] In our study, images passed once through the feature extraction network, which did not undergo training. Thus, the opportunity for these approaches or other image augmentations (flip / rotation / noise etc.) was lost. Augmentation and tests of generalizability will be foci of future work.

Conclusion

In conclusion, we developed models to identify placental parenchymal lesions using un-annotated whole slide images.

• A machine learning model can identify macroscopic placental lesions in whole slide images

• Attention maps made by machine learning models highlight diagnostic areas of interest.

• When models are wrong, attention maps can help interpret the model results.