A new deep learning framework to process Matrix-assisted Laser Desorption/Ionisation Mass Spectrometry Imaging (MALDI-MSI) data of Tissue Microarrays (TMAs)

Abstract

Matrix-Assisted Laser Desorption Ionization mass spectrometry imaging (MALDI-MSI) is a mass spectrometry ionization technique that can be used to directly analyze tissues and has led the way in the development of biological and clinical applications for imaging mass spectrometry. One of its advantages is measuring the distribution of a large number of analytes at one time without destroying the sample, making it a useful method in tissue-based studies. However, analysis of the MALDI-MSI images from tissue microarrays (TMAs) remains less studied. While several automated systems have been developed for tissue classification (e.g., cancer vs non-cancer), they process the MALDI data at the measuring point level, which ignores spatial relationships among individual points within the tissue sample. In this work, we propose mNet, a new deep learning framework to analyze MALDI-MSI data of TMAs at the tissue-needle-core level to ensure that the samples maintain their original spatial context. In addition, we introduced data augmentation techniques to increase data size which is often limited in biomedical data. We applied our framework to analyzing TMAs from breast and lung cancer. We found that our framework outperforms conventional machine learning methods in the challenging race detection task. The results highlight the potential of deep learning to assist pathologists in analyzing tissue specimens in a label-free, high-throughput manner.

1. Introduction

Matrix-Assisted Laser Desorption Ionization mass spectrometry imaging (MALDI-MSI) is an emerging technology that combines the analytical capabilities of mass spectrometry with microscopic information to investigate and understand molecular processes happening in specific cell types within a tissue.^1,2 It aims to increase the understanding of the molecular basis of diseases and augment observations of tissue morphology.² MALDI-MSI data not only contains a number of molecular distributions over the scanned tissue but also has the attributes of images with spatial information. However, the limited datasets and a lack of annotations hinder the development of classification models. Tissue microarray (TMA) is another innovative technology in the field of pathology as it allows high throughput analysis of multiple specimens at the same time. Specifically, a microarray contains many small representative tissue samples from hundreds of different cases assembled on a single histologic slide.³ Therefore, utilizing TMA with MALDI-MSI will provide access to spatial molecular distribution data from large cohorts of patients with rich phenotypic information.³

The structural hierarchy of MALDI-MSI data of TMA is demonstrated in Figure 1. Figure 1A is a microarray of tissue samples (cores). In the mass spectrometer, the tissues are raster-scanned, generating a mass spectrum for each measuring point (Figure 1B). Therefore, the TMA dataset includes a set of mass spectra associated with two coordinates, x and y. Based on spatial information, one TMA slide could be segmented into a set of cores with rich phenotypic information (Figure 1b). After MALDI-MSI, a set of resulting mass spectra across the entire surface of TMA slide is acquired. Each mass spectrum is a vector of intensities associated with m/z values (Figure 1d), while an ion image is obtained by extracting one m/z value and observing its intensity distribution across the TMA (Figure 1C). Both spatial and mass spectrum information helps us fully understand the tissue and identify potential biomarkers.

Figure 1:

Overview of MALDI-MSI data of TMA, from TMA to tissue core to a single spectrum. (A) Hematoxylin and eosin (H&E) staining of one TMA slide. The red box marks the tissue core shown in (a). (B) The scanned image of TMA slide by MALDI-MSI. Each green dot is a measuring point (resolution = 100um). The red box marks the tissue core shown in (b). (C) The selected ion image of TMA slide shows the intensity distribution of m/z 1809.6 after MALDI-MSI. The red box marks the tissue core shown in (c). (a)-(c) Magnified image of a core from one TMA slide. (d) A mass spectrum from a individual point shows a vector of m/z intensities after MALDI-MSI

The development in artificial intelligence has ushered in new opportunities for analyzing MALDI-IMS data of TMAs.^4–6 Existing approaches treat the MALDI-IMS images as a collection of measuring points and the classification step typically follows a two-step approach. In the first step, all classifiers were trained on and applied to each measuring point individually on the training and validation images. In the second step, each tumor core was assigned to the class receiving the most votes on the set of all associated points. The performance was measured by the number of correctly classified tumor cores on the (unseen during training) test datasets. Various classifiers can be used in the first step, such as support vector machine (SVM)⁷ and random forest.⁸ Recently, “deep learning systems” have been developed for learning features and classification in one step. Behrmann et al treated the points (mass spectra) in the cores of TMA were understood as one-dimensional images and utilized one-dimensional convolutional neural networks (CNNs) to classify the individual point (mass spectrum).⁶

While these approaches demonstrated promising results, they face three limitations. First, spatial information is not utilized in machine learning process. The prediction algorithm takes individual points or the tissue core as a whole with a vector of molecular features. Second, the composition of tissue samples even in a TMA is heterogenious. Although annotation of the precise tumor regions will significantly improve the classification accuracy, such annotation is fairly limited as it is labor insensitive and not irreproducible^6,9. Even analyzing tumor regions defined by pathologists, there will still be smaller stromal regions present in the tumor region due to the limitation of the co-registration analysis.⁷ Last but not least, automatic core region detection has not been well studied for MALDI-MSI. Though TMA core identification appears to be easy to the naked eye, it remains challenging to detect it automatically. To address these challenges, we propose mNet, a novel deep learning framework to directly analyze MALDI-MSI data of TMAs at the tissue-needle-core level. Our approach leverages both the molecular features of individual measuring points and their spatial relationships, to maximize classification accuracy. To separate the tissue core from TMA, we developed a new bounding box detection algorithm by exploring the molecular distribution of original core tissues. Finally, we utilized data augmentation techniques to increase the sample size, the limitation of which always hinders the use of deep learning on MALDI data. We applied our framework for TMAs of breast cancer and lung tumors and found that our framework outperforms conventional machine learning methods in a challenging phenotype identification task - race detection. The results highlight the potential of deep learning to assist pathologists in analyzing tissue specimens in a label-free, high-throughput manner.

2. Material and Methods

In this section, we discuss in detail our proposed framework. We will start with data collection and data preprocessing, followed by the bounding box detection algorithm. We will then describe our CNN-based model and several baselines.

2.1. Data collection

Breast cancer and lung adenocarcinoma arrays were constructed utilizing the 3DHistech TMA Grand Master instrument. Cases were culled from the VCU Anatomic Pathology backup database (2002-2020). Following pathologist slide review and marking of designated tumor areas, formalin fixed paraffin embedded (FFPE) blocks were then obtained through VCU Health Department of Pathology archives.

Two non-overlapping Triple Negative Breast Cancer (TNBC) arrays were constructed. TNBC TMA 01 contained 173 cores (18 row × 10 column layout size) and was comprised of 60 TNBC cases and 1 control case (2 normal liver cores). TNBC TMA 02 contained 221 cores (20 row × 12 column layout size) and was comprised of 73 cases and 1 control case (2 liver cores). The puncher diameter for each TMA was 1mm. Cases were cored in triplicate unless the size of available tumor from donor blocks was insufficient. Deidentified clinical annotation included length of follow-up period, histopathologic diagnosis, tumor site, age at diagnosis, race, gender, sentinel lymph node status, axillary dissection status, size of tumor, grade, hormone receptors status, Ki67, local/axillary recurrence status, neoadjuvant and adjuvant treatment details, metastatic disease status, and deceased status.

Two non-overlapping Lung Adenocarcinoma arrays were constructed. Lung Adeno TMA 01 contained 179 cores (20 row × 12 column layout size) and was comprised of 60 adenocarcinoma cases and 1 control case (2 normal liver cores). Lung Adeno TMA 02 contained 230 cores (20 row × 12 column layout size) and was comprised of 76 adenocarcinoma cases and 1 control case (2 liver cores). The puncher diameter for each TMA was 1mm. Cases were cored in triplicate unless the size of available tumor from donor blocks was insufficient. Deidentified clinical annotation race/ethnicity, gender, tumor site, age at diagnosis, histopathologic diagnosis, tumor site, tumor stage, metastatic disease status, neoadjuvant and adjuvant treatment details, deceased status, days from diagnosis to death, days from diagnosis to the last encounter, and smoking status.

Exported data sets for arrays and grids from the TMA Grand Master were associated with de-identified clinical data were provided through the Tissue and Data Acquisition and Analysis Core (TDAAC) and the Cancer Informatics Core (CIC) in collaboration with the Virginia Common University (VCU) Health Department of Pathology. This service was deemed non-human subjects research by the VCU Institutional Review Board (HM2471 and HM20004765 respectively). Sectioning was performed utilizing a Thermo Scientific Shandon Finesse E Benchtop Manual Rotary Microtome (#77500112) (Cheshire, United Kingdom) to produce slides of 4-micron thickness for subsequent staining. All data and slides for this study were analyzed anonymously.

Table 1 describes the pathological and clinical parameters of these datasets. Here we removed the tissue cores with missing data, thus the numbers are not consistent with the original TMA design. In total, we have 405 and 388 tissue cores for lung and breast cancers, respectively.

Table 1:

Characteristics of the TMA.

		Lung cancer # of cores = 405	Breast cancer # of cores = 388
Sex
	Male	159 (39.3%)	204 (52.6%)
	Female	246 (60.7%)	184 (47.4%)
Race
	BLACK	114 (28.1%)	208 (53.6%)
	WHITE	264 (65.2%)	159 (41.0%)
	OTHER	27 (6.7%)	21 (5.4%)
Age
	<50	63 (15.6%)	144 (37.1%)
	50-59	72 (17.8%)	115 (29.6%)
	60-70	162 (40.0%)	84 (21.6%)
	>70	108 (26.7%)	42 (10.8%)
	UNKNOWN	0 (0%)	3 (0.8%)

2.2. Data Preprocessing

In this study, we viewed the dataset with multiple ion images after MALDI-MSI as an image with multiple channels. Each pixel is the mass spectrum of a single measuring point. The channel is the intensity distribution of an m/z value over the TMA. The m/z value could roughly represent a type of molecule, such as a lipid, a glycan, and a peptide. Following the naming convention in the area of image processing, we call it a feature in this study. Therefore, a pixel is featured by a vector of molecular intensities, a core from one TMA slide contains multiple pixels, and the number for channels in a core equals the number of molecular features.

TMAs were scanned using MALDI-MSI following the steps detailed in Conroy’s work to capture the expression of glycogen and N-linked glycan¹⁰. The crucial role of glycans in mediating biological functions in both healthy and diseased state makes glycogen and glycan profiling a hot research topic. Multiple preprocessing steps are applied to the raw MALDI-MSI dataset to improve the signal-to-noise ratio and enhance image quality¹⁰. MALDI-MSI data files are first processed to adjust for mass drift during the MALDI scan using a carefully curated and established list of 50 MALDI matrix peaks (m/z) and 155 glycogen and N-linked glycan peaks (m/z). Matrix peaks in the feature set serve as the control molecules to normalize molecule levels between different mass spectra. Then molecular intensity is normalized by total ion count (TIC)¹¹ to decrease the variations among mass spectra. As a result, the model input feature set in this work is a 205D vector.

2.3. Bounding Box Detection

Figure 2 describes a new bounding box detection algorithm to crop the core regions. As shown in Figure 1, the cores are concatenated by each other, and their in-between distances are small, which makes it challenging for the sliding window detection algorithm since the detected bounding box may heavily overlap¹². Therefore, our method needs to handle this aforementioned problem. Specifically, we first whiten the core regions for each ion image by mapping the pixel value to 1 or 0. We then set up the bounding box to be 14 × 14 to include the whole single core region. Afterward, we apply a hierarchical sliding window detection which contains three rounds of pruning. The first round is to roughly select the bounding box regions where the area of pixel value 1 is greater than a threshold. In the second round of pruning, for each detected bounding box from the first round with its center coordinate, we crop a smaller rectangle of size 7 × 7 centered around its center coordinate and filter out all the candidates whose area of pixel value 1 is NOT equal to 7 × 7. In the third round of pruning, we want to ensure that the detected bounding boxes should not overlap each other. Therefore, we iterate all pairs of bounding box candidates and check if the distance between the centers of the candidate pairs is less than 14. If so, we consider them as overlapping and exclude one of the candidates from the pair. Figure 2 shows the pruning stages.

Figure 2:

Stages in detecting core regions. (A) Whiten the core regions(project pixel values into 1 or 0). (B) First round pruning. (C) Second round pruning. (D) Third round pruning (final round)

2.4. mNet and data augmentation

To fully establish the spatial position information and the core frequency value, we adopt a 2D CNN structure for the classification task. After obtaining the bounding box region for each core, we extract those cores individually and treat each 14 × 14 core image as a sample instance. In total, as shown in Table 1, we obtain 793 cores. the input dimension for CNN is 14 × 14 × 205, where 205 is the channel dimension.

In general, mNet comprises two convolutional blocks followed by one dense layer (Figure 3). Each block is composed of a number of filters (100 and 50 in our case). A convolution operation is performed between the input matrix and each filter, producing as output a new matrix. Then a max pooling layer is applied to reduce the amount of data to be sent to the next block and control overfitting. In the end, a feature flatten with a fully connected layer is performed, and a balanced cross-entropy loss is attached.

Figure 3:

The architecture of mNet.

Due to the high number of channels, the CNN architecture possesses a large number of parameters. We have specifically 557,650 trainable parameters in our CNN structure. The large parameter amount leads to easily overfitting when data is not sufficient. However, thanks to the unique CNN structure, we can apply data augmentation for compensating the insufficient and imbalanced training data^13,14. Specifically, we randomly rotated the images by 90, 180, and 360 degrees to create an augmented view. As a result, we have an additional 3 times more training data. Such an operation is crucial in increasing the diversity of the dataset and thus yielding effective and robust representations.

3. Results and Discussion

3.1. Race prediction

Many studies investigate tissue classification using the MALDI-MSI, but little is known about the correlation between race on MALDI-MSI when interpreting the images. Actually, there exist disparity in the molecular feature of the patient samples between races¹⁵. Our goal is to assess whether the molecular feature captured by MALDI-IMS is capable of distinguishing the race through the development of classification algorithm. For this purpose, we define the task of “race prediction” to evaluate if our model can predict self-reported races with MALDI-MSI alone as model inputs. While the ability to predict racial identity itself is not important, we want to emphasize that this capability is likely to be present in many MALDI-MSI models, provide the potential for racial disparities, and create risks for model deployment and patient triage.¹⁶

3.2. Experimental settings

We compared mNet with three baselines that use a two-step approach of feature extraction followed by a majority vote. Specifically, we first use the same bounding box detection algorithm to segment core regions. This is because these methodologies heavily rely on manually labeling the core regions. We then train classifiers on each measuring point using logistic regression, random forest, XGBoost.¹⁷, and a shallow feed-forward neural network with one dense projection layer. The input of these models is a feature vector of 205 elements. Finally, we conduct the majority vote algorithm for detecting the class of each core. To be more specific, after training the baseline classifiers, we classify every point within one individual testing core – that is every individual point votes for the class of tissue core, and the majority wins.

We split the entire dataset into 80% as training data and 20% as test data at the patient level. This ensured that no patient was in more than one group to avoid cross-contamination between the training and test datasets.

Our experiments report the per-class values of the F1-score, as well as their macro and micro averaging. Our MALDL-Net was implemented by Keras with a backend of Tensorflow. The proposed network was optimized using the Adam optimizer method. The learning rate was 5 × 10⁵. The experiments were performed on two NVIDIA GeForce RTX 2080 Ti GPUs.

3.3. Results and discussion

The performance of race prediction from MALDI-MSI is shown in Table 2. From the results, we conclude our findings as follows: Firstly, all the baseline methods could reach more than 65% macro F1 scores, indicating that our automatic bounding box detection algorithm could be successfully applied to the majority vote detection framework. Secondly, the CNN architecture with no data augmentation could have similar performances compared to the usual majority vote approaches (within 2∼3% F1 score difference), showing the correctness of using CNN architecture for detecting core features. Thirdly, after performing data augmentation, our CNN architecture could have a burst increase in prediction performance, and outperform all other baseline models (4% F1 score increment on average), indicating the great potential of applying modern CNN architectures in detecting core patterns.

Table 2:

Comparison of different model architectures in F1-score. mj - Majority vote

Models	white	black	micro F1	macro F1
(mj) Logistic regression	0.677	0.641	0.659	0.659
(mj) Random forest	0.676	0.588	0.654	0.645
(mj) XGBoost	0.711	0.647	0.684	0.679
(mj) Shallow neural network	0.632	0.613	0.623	0.622
mNet (w/o data augmentation)	0.683	0.621	0.631	0.624
mNet	0.724	0.714	0.693	0.693

Our results suggest that imputing CNN architecture with data augmentation could increase the performance of race detection. Meanwhile, our pipeline framework adapts an end-to-end learning scenario which shows effective and better performance. The bounding box detection part relieves the hard manually labeling task to a large degree.

4. Conclusions

In this work, we propose an end-to-end learning framework to process the MALDI-MSI data of TMAs. To the best of our knowledge, it is the first solution to incorporate spatial information for phenotype identification of TMA cores by using 2D CNN. In addition, the bounding box detection algorithm enables core regions to be automatically detected, which greatly facilitates high-throughput TMA experiments with rapid batch processing. Furthermore, we show that our framework achieves better classification accuracy compared with conventional pixel level-based classification. Our evaluation demonstrates a substantial accuracy gain by applying data augmentation for core-level classification.