Radiomic Models Predict Tumor Microenvironment Using Artificial Intelligence—the Novel Biomarkers in Breast Cancer Immune Microenvironment

Abstract

Breast cancer is the most common malignancy in women, and some subtypes are associated with a poor prognosis with a lack of efficacious therapy. Moreover, immunotherapy and the use of other novel antibody‒drug conjugates have been rapidly incorporated into the standard management of advanced breast cancer. To extract more benefit from these therapies, clarifying and monitoring the tumor microenvironment (TME) status is critical, but this is difficult to accomplish based on conventional approaches. Radiomics is a method wherein radiological image features are comprehensively collected and assessed to build connections with disease diagnosis, prognosis, therapy efficacy, the TME, etc In recent years, studies focused on predicting the TME using radiomics have increasingly emerged, most of which demonstrate meaningful results and show better capability than conventional methods in some aspects. Beyond predicting tumor-infiltrating lymphocytes, immunophenotypes, cytokines, infiltrating inflammatory factors, and other stromal components, radiomic models have the potential to provide a completely new approach to deciphering the TME and facilitating tumor management by physicians.

Introduction

For malignant tumor cells, the tumor microenvironment (TME) plays an important role in the growth and invasion of tumor tissues, which was realized early by researchers, and there are a great number of studies referring to the clinical importance of TME and related detection methods. The TME consists of stromal cells, infiltrating immune cells, inflammatory cells, endothelial cells, adipocytes, and fibroblasts, ¹ some of which can enhance the aggressiveness and growth of tumors; others may function as antitumor components that help sustain homeostasis and can be closely associated with prognosis, treatment efficacy or residual disease. Thus, learning about the TME can benefit the management of cancer patients and help clinicians choose better therapeutic strategies.

However, to obtain information on the TME, invasive operations, such as biopsy, must be performed to obtain tumor tissues, which are then sent to pathologists for staining and observation under light microscopy. The whole process can be time-consuming and cause a certain degree of damage and related complications to patients. Most importantly, it is impossible to supervise dynamic changes in the TME in every region of tumors, as biopsy is usually performed only once before surgery or chemotherapy.

Radiological imaging was only a routine tool for monitoring cancerous lesions in patients and assessing efficacy without requiring more applications; nevertheless, with the development of radiomic techniques, radiological images from magnetic resonance (MR), computed tomography (CT), positron emission tomography-CT (PET-CT), or even ultrasonography are increasingly translated into numerous data that can be modeled through various algorithms and therefore help clear issues clinically. We call this technology “radiomics.” Since its introduction by Lambin et al in 2012, an increasing number of retrospective and prospective studies on tumor radiomics have been conducted, varying from predicting malignancy, prognosis, subtypes, and therapy response. The evaluation of relationships between the TME and radiomic profiles has also received more attention in solid tumors such as breast cancer, non–small-cell lung cancer, gastric cancer, and hepatocellular carcinoma in recent years. Especially in breast cancer, radiomics has been applied in several aspects in numerous studies, especially in the past 5 years, including investigations into predicting benign or malignant breast tumors, axillary lymph nodes before operation, reaction to neoadjuvant chemotherapy, molecular subtypes, prognosis of both early- and late-stage patients, and constitution of the TME. However, further validation is needed, particularly for TME-radiomic models since there is a lack of sufficient prospective patient data, and the interpretability and general applicability of radiomics are also deficient as biomarkers.

Breast cancer is the most common cancer among women and can pose a serious health risk, particularly in advanced-stage patients. As immunotherapy shows more potential antitumor implications in breast cancer and depends heavily on immunocytes surrounding tumor cells, new noninvasive approaches are urgently needed to assess the TME. Radiomics or radiogenomics (a combination of radiomics and genomics), as well as approaches that incorporate clinical, pathological, and serum features, have shown promising results as novel biomarkers for deciphering TME in numerous studies. In this article, we summarize the current state and progress of radiomics in inferring breast cancer TME components and provide a brief overview of future directions.

Macroscopic Image Features Reflect Microscopic Microenvironment Features

Images derived from techniques such as CT or MR are formed from a collection of pixels or voxels that convey information about the size, shape, location, and texture of tumors, and radiologists usually use this information to make a judgment about disease status. However, it is hardly possible to extract more precise and abstract information from images in depth with the naked eye, and the information we can obtain is always qualified but not quantified. With the assistance of artificial intelligence (AI) hosted by computers, radiomics is a high-throughput method that can extract hundreds or even thousands of direct and extended features, such as statistical features, from images, translating them into vast amounts of data that can provide more precise and detailed information to physicians. ²

Progressive Image Technologies for Radiomics

Most radiomic studies of the breast cancer TME are based on MR imaging especially dynamic contrast-enhanced imaging (DCE-MRI). This imaging odality monitors functional features of local tumors in the breast, measuring properties of tissue microvasculature, and has validated capacity for cancer detection, diagnosis, staging, and assessment of treatment response. ³ PET-CT and single-photon emission CT (SPECT) can detect functional features of tumor tissues as well and have already been investigated in relation to the breast cancer TME. ⁴ In addition, normal breast imaging modalities such as mammography and ultrasonography are widely utilized in breast cancer radiomic studies, although CT is rarely adopted. For example, a 2017 study showed a relationship between tissue malignancy and image feature-density fluctuations of pretumor tissues using high-quality full-field digital mammography. ⁵ In addition to this, multimodal modalities (referring to a combination of several disparate methodologies or different imaging sequences) are also applied in varying sorts of radiomic studies in breast cancer or other solid tumors including gastric cancer, glioma, etc. For instance, a 2022 study utilized both (18)F-fluorodeoxyglucose (FDG) PET and MRI images to build radiomic models to discriminate between benign and malignant breast lesions, and another recent study used CT and MRI to predict primary tumors for brain metastasis.^6,7 Both studies showed that multimodal models perform better than mono-modal ones. Additionally, multisequence imaging modalities are the most common in MRI-based radiomics, which will be highlighted in the following sections. Multisequence ultrasonography has also been adopted in some studies. For example, a study by Xu et al ⁸ in 2022 involved examination of five regions of breast tumors on B-mode ultrasound and contrast-enhanced ultrasound images and showed that multiimaging sequences of multiregion radiomic models exhibited higher area under the curve (AUC) values than those based on single sequencing of a single region. Other novel radiological techniques such as PET–MRI and photoacoustic imaging also show the capacity to detect breast cancer and may have great potential in radiomic studies for breast cancer, as they can convey enrich metabolic information in tumor cells and the TME.^9,10.

In conclusion, the most promising and generally adopted imaging modalities for predicting TME are those that, like PET-CT, multiparametric MRI, and DCE-MRI, can detect metabolic features in the microenvironment.

Various Forms of Artificial Intelligence in Workflow of Radiomics

AI technologies include machine learning, neural networks, and deep learning. Actually, machine learning incorporates neural networks, and neural networks incorporate deep learning in an increasingly complex computing direction. The development of AI has clearly promoted the progression of radiomics. AI works throughout the radiomic workflow, including the selection of regions of interest (ROIs), segmentation of ROIs, extraction of features, features processing and analysis, and model validation.^11,12

Artificial Intelligence in Delineation of ROIs

Delineation of ROIs can be conducted by manual operation by expert radiologists. However, this is not proposed now since it can be affected by the experience of radiologists and also it demands much time if there is a huge amount of data. Thus, benefiting from the progression of AI algorithm, (semi)automated modalities are generally applied. These modalities include semiautomated methods (thresholding-based and region-based segmentation approaches), automated methods (statistical pattern recognition, c-clustering), more complex atlas- and rule-based segmentations, modalities based on deep learning algorithms, and even combinations of several modalities. Advanced segmentation deep learning algorithm network structures can include 2.5D convolutional neural network (CNN), three-dimensional CNN, fully convolutional network), U-Net, convolutional residual networks, etc. ¹³ While these computer-assisted tools greatly decrease the time and energy cost saving, they can easily lead to mistakes, especially in processing low-quality or complex images; thus, it is commonly suggested that manual modification be performed after (semi)automated processing.^11,12,14

Artificial Intelligence in Features Extraction and Selection

The most pivotal section of radiomics is feature extraction and analysis, in which enormous amounts of data need to be mined. Therefore, AI analysis tools are necessary for this high-throughput nature. Radiomic features are classified into semantic features and agnostic features. Semantic features such as shape, size, location, and necrosis are visual and always used by radiologists to depict ROIs; qualitative features, however, still play a critical role in modeling. Agnostic features, usually called “radiomic features,” are thought to be mathematical and quantitative and are extracted through computed algorithms. They are further classified into shape features, first-order statistical features, second-order statistical features, and higher-order statistical features, such as kurtosis, skewness, wavelets, and fractal dimensions. All features mentioned above are based on traditional machine learning models. As for features extracted from deep learning models, they are more abstract and agnostic.

After extracting large-scale features, whether beneficial to modeling or not, further selection of these features should be conducted manually or through machine learning. The purpose of selecting features is to avoid irrelevant or redundant features, which may cause overfitting of the model or other issues that may have implications for model performance. In addition, archetypal features are then obtained through dimensionality reduction techniques. Therefore, algorithm methods including cluster analysis, principal component analysis, nested architecture analysis such as LASSO regression and recursive feature elimination can help process these features before modeling. ¹⁵

Artificial Intelligence in Building Model

Only when feature data is processed does modeling begin based on AI technologies including machine learning, neural networks, or deep learning. Among them, machine learning is most widely used in radiomics and can be divided into unsupervised, semisupervised, supervised, and reinforcement learning. Detailed algorithms can include decision-making tree, support vector machine, random forest, logistic regression, k-means clustering and more complicated ones including CNNs and deep belief nets. In fact, using several kinds of algorithms simultaneously is allowed if it can result in better performance. Aside from image feature data, external information such as gene mutations, clinical features, and pathological features can also contribute to the model. In radiomic studies about the TME, radiogenomics is usually applied, as there is also a tight relationship between the TME and the genome.

Artificial Intelligence in Validation

Model validation can be vital to complete a robust model, and using an intravalidation cohort, an intervalidation cohort, and an external validation cohort to validate the model is always implemented in studies with abundant patient data. For those without enough data, however, it is also a feasible way to employ machine learning models to make internal validation techniques, such as cross-validation (CV), especially leave-one-out CV and nested CV.^14,16

Artificial Intelligence in Digital Pathology

In studies of radiomics to predict TME, reading pathology information from tumor by pathologists is also a key but time-consuming work, hence it's a need for assistance by computer like digital pathology. Digital pathology includes the process of digitizing histopathology slides using scanners and the analysis of these digitized whole-slide images.^{. 17} With the employment of AI–machine learning and deep learning models, digital image features from tumor and surrounding TME are detected and classified (like fibroblast and matrix features, vascular features, lymphocytes features, and even high-order TME features). That also helps reveal intratumor heterogeneity based on spatial quantitative analysis. ¹⁸ More standard and detailed pathological information by AI always welcome in radiomics studies.

Radiomic Studies Regarding the TME Reveal Crucial Biomarkers for Breast Cancer

With regard to employing radiographic modalities to predict features of the TME, PET-CT has already been used to show the relationships between tracer agent and tumor-infiltrating lymphocytes (TILs). For example, a recent retrospective study incorporating different subtypes of breast cancer used a Cox proportional hazards model to analyze tracer-FDG uptake (maximum standardized uptake value [SUVmax]) and TILs. The results showed that a high SUVmax was significantly related to a high level of TILs and to recurrence-free survival. Several other studies have reported similar results. However, these studies cannot be defined as radiomic studies since there was no complicated image feature selection or analysis considered high-throughput, and the only feature was hypothesized in advance.^19–22 The study mentioned above in earlier years used mammography images to detect features related to the malignancy of breast tumors in numerous imaging data through a method called two-dimensional Wavelet Transform Modulus Maxima. This approach could be considered a radiomic approach.

Tumor-Infiltrating Lymphocytes Have Become More Crucial in the Immunotherapy Era

With an essential role in the TME, TILs, especially ones in the stroma, have been evaluated in breast cancer to predict prognosis and the response to neoadjuvant chemotherapy. A pooled analysis of 3771 patients with different breast cancer subtypes in 2018 showed that TILs are positively correlated with the response to neoadjuvant chemotherapy, but their relationship to prognosis varies according to the specific subtype. ²³ Moreover, with the immune checkpoint inhibitor agents atezolizumab and pembrolizumab successively approved by the FDA for triple-negative breast cancer (TNBC) of advanced or even early stage, there is increased focus on the TILs’ capacity to predict benefit from checkpoint inhibitors in patients. ²⁴ In addition, since TILs are often analyzed in randomized phase trials of PD-1 or PD-L1 inhibitors, a confirming conclusion could be made that high-level TILs suggest a better response to checkpoint inhibitors, similar to the evaluation of PD-L1 protein on TILs.^25,26

To identify TILs, usually, histological slides of tumor tissue are indispensable, and a common method is to use hematoxylin and eosin (H&E) staining, immunohistochemical (IHC) staining or even gene-expression analysis such as RNA sequencing. However, these methods cannot avoid the disadvantages of localization of tissues from biopsy, lack of dynamic information, and other difficulties in manual observation of slides. Actually, radiomics can complement pathological observation by providing a full-scale and dynamic view. ²⁷ In addition to TILs, other immune cells such as macrophages play a crucial role in immune depression and can regulate PD-1/PD-L1 expression. ²⁸ Nonimmune cells including fibroblasts are also an important component of the microenvironment. They can also be reflected by imaging features. ²⁹

Biomarkers in Different Scale Evaluate Tumor Microenvironment

Current method for the evaluation of TME can incorporate traditional histopathological observation, peripheral blood test, related gene analysis and radiomics. The most direct and explicit method to evaluate tumor immune microenvironment (TIME), of course, is the histologic observation through IHC and even multiplex IHC/immunofluorescence (mIHC/IF). ³⁰ Though this method is enough straightforward and easily realized, considering the inconvenience, TME heterogeneity and the bias by pathologists, ³¹ it's not an ideal method. The peripheral blood may also reflect TME to some degree. For example, a study in 2020 found that expanded clonotypes CD8 + T cell found in the tumor and normal adjacent tissue can also typically be detected in peripheral blood and they were derived from peripheral blood. ³² Another study also found the correlation between circulating neutrophil expansion and local inflammatory milieu in tumor. ³³ However, this method only take a few specific tumor types into consideration while TME include all kinds of cells or molecule. In addition, with the rise of second-generation gene sequencing and single-cell sequencing, the TME can also be evaluated by analyzing gene expression profiles like a model based on differently expressed genes. ³⁴ Brief introduction of these approaches can be seen in Figure 1.

Figure 1.

Four types of approaches to evaluate tumor microenvironment (TME). They include pathological observation, peripheral blood detection, radiomics, and genome analysis. (Figure 1 is created with BioRender.com and has obtained confirmation of publication and licensing rights).

Compared with the three methods mentioned above, novel biomarkers gained from radiomics are more mathematical and abstract. Radiomics is a kind of data-driven and high-throughput tool and thus it requires a larger amount of data from images of whole tumor and even whole human body. In one way, benefiting from this feature, more precise and comprehensive information is dug and this help overcome heterogeneity of TME in space and time to some degree. In another way, biomarkers from radiomics are generally considered lack of clear biological meaning while the other three methods all have biological meaning to some extent. That is like a black box—radiomics is inherently hard to elucidate the biological underpinnings of the observed relationships, especially, for the agnostic features extracted from images. The trend to pursue powerful prediction ability regardless of biological explanation in radiomics studies may hinder the development of radiomics. ³⁵

Now through detecting the relationship between radiomics and gene expression, immunohistochemistry, local pathologic analysis, or habitat imaging, underlying biological meaning is indirectly and partly explained. ³⁵ However, more studies for exploring more clear biological meaning are still urgently needed.

Radiomic Studies Predicting the Breast Cancer TME in Recent Years

In 2018, a study led by Sun et al ³⁶ used a combination of contrast-enhanced CT images and RNA-seq genomic data to detect infiltrating CD8 + T cells in solid tumors including breast cancer (only 17 breast cancer cases in the training cohort). It may be that this was the first study to investigate the correlation between TILs and a radiomic signature related to breast cancer. Most radiomic studies investigating breast cancer TME (referred components can be TILs, immunophenotypes, various cell abundances, etc) were published between 2021 and 2022 and now, and publication of such literature is likely to increase more dramatically. About half of studies involved data from The Cancer Imaging Archive (TCIA) database and The Cancer Genome Atlas (TCGA) database while some were based on the data from other open-source or non-open-source databases. However, as aforementioned, nearly all studies were retrospective, rendering the validation insufficient, despite the inclusion of many cases in several studies. Figure 2 shows brief simulation of radiomics studies for predicting TME and Table 1 shows a summarization of related studies from 2018 to 2022.

Figure 2.

Summary of TME-predicting radiomic building process. TME-predicting radiomic model can evaluate various components (including neutrophil, NK cell, lymphocyte etc) through analysis of correlation with radiological images features (shape features, histogram features, texture features and wavelet-transformed features etc). Abbreviations: TCIA, The Cancer Imaging Archive; TME, tumor microenvironment; NK, natural killer cells. (Pathological and radiological images in Figure 2 are cited from the open-access database TCIA. ⁴⁶ Parts of cell pictures used in Figure 2 from Servier Medical Art. Servier Medical Art by Servier is licensed under a Creative Commons Attribution 3.0 Unported License (https://creativecommons.org/licenses/by/3.0/)).

Table 1.

Radiomic Studies of the Breast Cancer TME From 2018 to 2022.

Publication date	Main researcher	Purpose	Imaging modalities	Included cases	Patient characteristics	Modeling modalities	AUC value	Main results
2018 36	Roger Sun	1.Preditct infiltrating CD8 + T cells 2.Immunophenotype 3.Immunotherapy	Contrast-enhanced CT	n = 135 solid tumors (17 breast cancer in training cohort)（0 breast cancer in validation cases）	Uncertain	Elastic-net regularized regression	Training set:0·74 (95% CI 0·66-0·82; p < 0·0001) validation set:0·67 (95% CI 0·57-0·77,p = 0·0019)	The radiomic score was significantly higher in the immune-inflamed group than in the immune-desert group
2019 37	Nathaniel Braman	1.Predict peritumoral lymphocytic distribution 2.Discriminate the her-2 positive patients 3.Neoadjuvant chemotherapy efficacy	DCE-MRI	209(27 were used to build a radiomic-TME model)	12 HER2-positive 15 Non-HER2-positive	3-fold, Cross-validation setting, Bhattacharyya distance	None	Gabor features show a spatial association between reduced expression and dense lymphocyte distribution
2021 38	Wen-jie Tang	<10% stromal TILs as low TIL levels versus≥ 10% stromal TILs as high TIL levels	T1 W T2, DWI, DCE	133(Training (n = 92) Validation (n = 41))	Luminal:92, Nonluminal:41	LASSO， logistic regression	Training set:0.950 validation set:0.870	DCE phase_6 is the most important factor
2021 39	Tiantian Bian	Low TIL (<50%) versus high TIL (≥50%) subgroups	T2, T1, DWI, DCE	154(a training cohort (N = 87) and a test cohort (N = 67))	Luminal A:46 Luminal B:65 TNBC:32 HER2 positive:11	mRMR, LASSO, logistic regression	Training set:0.88 Validation set:0.83	Wavelet features are the most important radiomic features.
2021 29	Dooman Arefan	Relationships between radiomic features and cell-type abundance in TME	DCE-MRI(T1 based)	73(training:43, validation:30)	ER positive:61 PR positive:64 HER2 positive:8	Linear regression, recursive feature Elimination, Binary logistic extreme Gradient boosting framework	Intravalidation: ranging from 0.5 to 0.83 Intervalidation: ranging from 0.5 to 0.68	Models with notably higher AUCs include the models predicting NK cells and neutrophils
2021* 40	Yunfang Yu	1.predict preoperative axillary lymph node metastasis 2.tumor microenvironment relationship with radiomic features	DCE-MRI(T1 + C,T2)	n = 90(for tumor microenvironment exploration)	uncertain	random forest algorithm	None	radiomic features are linearly correlated with immune cells, such as M0 macrophages, B naïve cells, and neutrophils
2021 41	Nina Xu	low TILs (< 10%) versus medium-high TILs (≥ 10%)	T2, T1, DWI, DCE	172(training 121 and testing 51)	ER positive:51 PR positive:58 HER2 positive:62	LASSO, linear regression, random forest algorithm	training set: 0.800 (95% CI, 0.709–0.892) validation set: 0.842 (95% CI, 0.730–0.954)	The combined prediction model outperformed the clinical feature model
2022 42	Xiaorui Han	1.predict immunoscore for breast cancer (ISBC) 2.use ISBC to predict prognosis	DCE-MRI	1. n = 335 for calculation of immunoscore 2. n = 120 for radiomic model 3. n = 155 for evaluation of prognosis prediction of radiomic-based ISBC	ER positive:99 PR positive:88 HER2 positive:13	Random forest machine-learning	Training set: 0.899 validation set:0.815	Prognosis is significant between high-ISBC and low-ISBC based on radiomics
2022 43	Guan-Hua Su	1.< 20% stromal TILs versus ≥ 20% stromal TILs 2.transcriptomic association with radiomic scores	T2, T1, DCE	139(training cohort (n = 98) and validation cohort (n = 41))	All TNBC	Elastic net and logistic regression	training set: 0.868(95% CI 0.797-0.938) validation set: 0.790(95% CI 0.638-0.943)	1.wavelet-related features are mainly selected 2.transcriptomic features are different between high and low TIL cases based on radiomics.
2022* 44	Xuanyi Wang	1.predict the prognosis of patients with locally advanced breast cancer 2.heterogeneity of TME between radiomic score groups	T1-enhanced MRI	278(training (n = 139) validation (n = 139) external validation(n = 91))	ER positive:90 PR positive:77 HER2 positive:96	LASSO regression	None	1.The high-score group showed a significantly lower MHC molecule score. 2.Activated NK cell levels were higher the in low-score group
2022 45	Seung Hyuck Jeon	1.predict immunophenotypes 2.neoadjuvant chemotherapy and radiomic scores	DCE-MRI	1.Radiomic model building cohort(n = 182)(training (n = 137) and validation (n = 45) cohorts) 2.Neoadjuvant chemotherapy cohort(n = 64)	Luminal A:72 Luminal B:75 TNBC:17 HER2 positive:18	LASSO regression	1.(RM-wholeFC) training, AUC = 0.973; validation, AUC = 0.985 2. (RMperiFC), training, AUC = 0.993; validation, AUC = 0.984	1. The accuracy of the combined radiomic model was 0.911 2. Tumors exhibiting complete response had significantly higher scores from RM-wholeFC and lower scores from RM-periFC

Abbreviations: TME, tumor microenvironment; TNBC, triple-negative breast cancer; DWI, diffusion-weighted imaging; RM-whole_FC, radiomics model based on all features of whole tumor; AUC, area under the curve; MHC, major histocompatibility complex; CT, computed tomography; DCE-MRI: dynamic contrast-enhanced magnetic resonance imaging.

“*”means radiomic models in these studies are indirectly related to the tumor microenvironment of breast cancer.

Radiomics Study Design for Predicting Breast Cancer TME

The common design in these studies is that the included patients are randomly divided into cohorts of training and validation with or without an external validation cohort. The external validation cohort is always from the TCIA database. Moreover, nearly all studies are based on DCE-MRI with varying sequences including T1W1, T2W1, diffusion-weighted imaging (DWI), and especially DCE sequences with different phases. Exact features extracted from DCE sequences show high correlation coefficients with features from TME analysis in nearly all DCE-MRI-based radiomic studies for predicting breast cancer TME. Furthermore, for the evaluation of the TME, conventional H&E-stained slides are observed by expert pathologists using microscopy, and data from databases such as TCGA can also be used to conduct gene-expression analysis algorithms to obtain information on the abundance of microenvironment cell types. Afterward, to compare distinctive TIL levels, a simple standard is adopted for division (defining < 10% portion of stromal TILs (sTILs) as a low group, 10%–50% stromal TILs as an intermediate group, and > 50% as a high group) in several studies. ⁴⁷ However, this classification method has not been confirmed to be valid, and there is a lack of standard cutoff values for TILs. Different studies about radiomic TILs in breast cancer adopt only 10% or 50% as the standard to divide patients into low and high groups, which may introduce difficulties in comparing results in these studies. Additionally, the results based on manual observation can cause variation, thus computed pathology is getting more attention now. For example, in the study conducted by Wang et al, ⁴⁴ computed pathology was used to show tumor cell differentiation and tumor morphology and evaluate the relation of radiomic scores to them. In addition, in several studies, significant differences in baseline characteristics between low- and high-TIL groups were shown, making validation of the results weak.^38,39

Key Radiomics Features Found for Predicting Breast Cancer TME

Since it is considered that a generally used model should comprise limited features and since overfitting should be avoided, in the extraction and selection of features, most studies involve choosing 5–15 key features to build the model, while the study by Han et al ⁴² adopted 21 features. As mentioned above, features from the DCE sequence seem to contribute mostly to the founding of radiomic signature, and the study by Tang ³⁸ even shows that the features from delayed phases of DCE sequences, especially DCE_P6, have more powerful prediction abilities. When referring to specific features, second-order and higher-order features are preferred. Second-order features such as gray-level co-occurrence matrix, gray-level run-length matrix, and neighborhood gray-tone difference matrix are generally selected in TME-radiomic studies. Higher-order features are preferred to be acquired in these studies through wavelet transformation of these features using a wavelet filter. These transformed features can show an intensity level of voxel spatial distribution and are sensitive to tumor heterogeneity. Actually, regardless of whether the purpose is to predict prognosis, nonchemotherapy response, or the TME in breast cancer or other solid tumors, second-order and higher-order features are always the majority of key features.

Predicting Spatial Heterogeneity in TME by Radiomics

The spatial distribution of TILs was also correlated with radiomics in another study by Jeon et al in 2022. According to the immune profiles classification proposed by Chen et al ⁴⁸ in 2017, immune-desert profile could be simply seen as TILs infiltrating tumor parenchyma, immune-excluded as in tumor invasive margin, inflamed as in none area. Then, Jeon et al had the ingenious idea that radiomics models could be categorized according to two purposes: One would be to use DCE-MRI imaging features of whole tumor area to detect CD8+ T cells to distinguish inflamed and noninflamed phenotypes, and another would be to use peri-tumor phenotypes to distinguish immune-desert and immune-excluded phenotypes. Both models are built by, respectively, exploiting four sequences of DCE and combination models using all features from sequences or four sequence model radiomic scores. The results showed that both radiomics model based on all features of whole tumor (RM-whole_FC) and the radiomics model based on all features of peri-tumor (RM-pri_FC) unfolded higher AUCs than the respective cohorts, although the AUC of RM-pri_FC was not significantly higher. Another result of this study is that when the two aforementioned models were applied to predict the response of neoadjuvant chemotherapy in another cohort, RM-whole_FC score was higher in patients with ypCR, while RM-pri_FC score was higher in patients with nonypCR, which is not consistent with previous studies that showed that higher levels of CD8 + T cells were predictive of better response to neoadjuvant chemotherapy. This result may show potential implication of spatial distribution of immune cells. In addition, then the combination of RM-whole_FC and RM-peri_FC to predict phenotypes finally showed the accuracy of 0.911 in the validation cohort. This study supplied a disparate method to the analysis of tumor region imaging, which can be segmented into different areas to be more accurately and further analyzed. ⁴⁵ Actually Braman et al ³⁷ earlier in 2019 also modeled using peripheral tumor (0-3 mm closest to tumor) DCE-MRI imaging features of breast cancer to predict TILs and peritumoral lymphocytic density. An interesting result showed that the greater the distance from the tumor, the tighter relationship was observed between radiomics features and peripheral lymphocytic density although only 13 cases were included. In addition, the result from another cohort (n = 117) in this study demonstrated that features from the peritumoral regions had more powerful discrimination of Her-2 enriched patients than features from intratumor.

Multimodal Models Extend Prediction Capacity

In addition to radiomic features, for the purpose of developing better performance in terms of predicting the TME, significant clinicopathology features are usually selected to build a radiomic nomogram through multivariate regression. Molecular indices including Her-2 status, hormone receptor status or Ki-67 are always incorporated.^38,39,41,44 Actually, it has been already realized that Her-2-positive and TNBC subtypes of breast cancer show a higher level of TILs than the luminal subtype. However, all three studies based on models combining radiomic features and clinical features showed that no significant difference was found between radiomic models and nomogram models, although AUCs of nomogram models were all higher than those of radiomic models.^38,39,41 In addition, the nomogram also shows that clinical features contribute little to the model, suggesting that radiomics can have a much greater potential for predicting TME than clinical features. Nevertheless, clinical models are more universal, while radiomic models now seem to lack reproducibility when applied in other situations.

In addition, multiomics, such as the combination of radiomics and genomics or radiomics and transcriptomics, is getting increasingly popularized, among which the most focus is radiogenomics. Actually, radiogenomics can include two types of studies: one that investigates the relationship between imaging features and genomic features, and another that explores the potential of predicting prognosis or therapy efficacy through the combination of both. For instance, in the study by Fan et al ⁴⁹ in 2022, a radiomic model was built to predict prognosis of breast cancer using the Oncotype DX Recurrence Score, and another study by Woodard et al ⁵⁰ showed the relationship between BI-RADS-based features and the Oncotype DX Recurrence Score. Additionally, another study by Chen et al ⁵¹ in 2022 showed that models combining radiomic and genomic features performed better than radiomic models or genomic models alone in predicting ALNM of breast cancer. Radiogenomics has been applied to predict the TME in several types of solid tumors but not breast cancer.^52–54 For example, in the study by Feng et al in 2022, a genomic model was built to predict the hypoxia pattern of patients with ovarian cancer. Additionally, different hypoxia risk groups identified based on genomics showed differences in the TIME, prognosis and chemotherapy efficacy. In addition, radiomics based on CT was adopted to predict different hypoxia risk groups. However, the AUC of this radiogenomic model was only 0.703 in the validation cohort. ⁴⁵ In this study, a radiomic model that directly predicted hypoxia patterns was not built, and thus the comparison between radiomic models and radiogenomic models is not clear. Moreover, no radiogenomic models combining both imaging and gene features have been built to predict the TME. As for breast cancer, radiogenomic models are primarily focused on predicting subtypes and recurring related genes based on MRI, while studies on the TME of breast cancer are lacking (the radiogenomic concept is shown in Figure 3).^55,56

Figure 3.

Concept of radiogenomics. There are two types of radiogenomics studies. One is to build the relationship between radiological images and genome. Another is the prediction of TME (for example) through combination of radiological images and genome. Abbreviations: TCIA, The Cancer Imaging Archive; TME, tumor microenvironment. (Radiological images in Figure 3 are cited from open-access database TCIA ⁴⁶ ).

In three studies referred to as radiomic studies of breast cancer, a relationship was also shown between transcriptomics and radiomics.^40,43,44 Differentially expressed genes (DEGs) were analyzed using KEGG pathway and GO analysis, and the results showed that genes related to metabolic, immune functions, and cytokine pathways were significantly different in low and high radiomic score groups (different radiomic score groups in tree studies were divided according to the TIL level, axillary lymph node status and prognosis, respectively). For example, the study by Su et al found that the gene expression level associated with PD-L1 expression and the PD-1 checkpoint pathway was significantly higher in the high radiomic score group (the group predicting higher levels of TILs). This result demonstrates the consistency of radiomic models as they related to transcriptomic levels. In addition to DEGs, long noncoding RNAs are also found related to radiomic features.^57,58 For instance, the study by Yu et al ⁴⁰ also showed the heterogeneity of long noncoding RNAs and types of methylated sites in 90 cases across two radiomic score groups. Overall, the multiomic approaches seem to have higher performance grades in various models.

Radiomcis Models for Evaluating Multicomponent TME

Abundance of Various Cells in TME

In addition to providing an overview of TILs, radiomics can detect specific immune cells such as CD8+ T lymphocytes, macrophages, other stromal cells, including fibroblasts, and even key molecules. The study by Arefan et al ²⁹ in 2021 investigated both the univariate and multivariate analysis relationships between radiomic features and the abundance of 10 types of cells in the TME of breast cancer, respectively. The results of univariate analysis showed that visual shape features including diameter and perimeter were positively correlated with neutrophil abundance while kinetic features were more correlated with fibroblasts and endothelial cells. In multivariate analysis, abstract features such as the mean quick and intermediate PE are more likely to be adopted in models, while tumor volume was not useful in distinguishing the abundance levels of the 10 types of cells. This is consistent with the conclusion that abstract features occupy an important position in radiomics. However, it should be noted that this study only used a training cohort of 43 patients to predict 10 types of cells, which may introduce significant randomness. Additionally, another study by Su et al ⁴³ in 2022, including 139 breast cancer patients, showed differences in immune cell abundance in the TME between high and low radiomic-TIL score groups. In addition, the result showed that besides T and B lymphocytes, other immune cells such as M1 macrophages, activated NK cells, and plasma cells are also more enriched in the high score group. The results of the above two studies indicate a tight relationship between manifold cell types and imaging. However, some cells such as macrophages can function distinctly and block the function of TILs and accelerate tumor progression ²⁸ ; thus, further comprehensive radiomic models based on multiple types of cells can be developed to more precisely predict prognosis, subtypes of disease, therapy efficacy, etc.

Molecules in TME

To further realize the potential biological mechanism, Su et al also distinguished molecules in the TME between the low and high radiomic-score groups. These molecules included cell surface molecule such as costimulators, coinhibitors and major histocompatibility complex (MHC), and cytokines including interleukins, colony-stimulating factors (CSFs), interferons and chemokines, all of which may describe the malignancy or immune-escape abilities of tumors and that may assist clinicians in making management decisions more beneficial to patients. In addition, in a recent study on head and neck squamous cell carcinoma (HNSCC) by Wang et al, ⁵⁹ the scholars constructed a radiomic model using CT images to predict the mRNA expression of CD27. CD27 is a member of the tumor necrosis factor receptor superfamily, which is vital to the functioning of both T and B lymphocytes. Thus, the study also found that it could influence the infiltrating immune cells, and it was related to the prognosis of patients with HNSCC.

Radiomics-Based Tumor Immune Microenvironment Signature Predict Radioimmunotherapy Efficacy

As realized in some studies, radiotherapy and TIME have a mutual effect on each other. Radiotherapy can activate modulate tumor phenotypes, enhances antigen presentation and tumor immunogenicity, increases production of cytokines, and alters the TME, enabling the destruction of the tumor by the immune system. In turn, activated immune cells and factors will help boost the efficacy of immunotherapy. ⁶⁰ Consequently, immunotherapy is generally thought to be combined with radiotherapy to enhance therapeutic effect. And thus the comprehensive detection of TIME can be vital in such therapeutic strategy.

To describe TIME to predict response effect, radiomics signature may also be adopted . A study by Sun et al ⁶¹ applied the radiomics-based CD8-T cells score on multiple lesions on the baseline CT scan of 94 solid cancer patients treated with immunotherapy and radiotherapy and found the score at baseline of responding lesions was higher than those of stable or progressive lesions (AUC = 0.63, 95%CI 0.56 to 0.71), p = 0.0020). Korpics et al also had a similar conclusion (HR = 0.18; 95%CI 0.04 to 0.74; p = 0.018).

Although radiotherapy is not major care in early-stage breast cancer, it's significant when there is a cerebral metastasis or skeletal metastasis. Meanwhile, immunotherapy is gradually functioning in advanced breast cancer. Therefore, a radiomics-TME biomarker can have a further application in such similar combination therapy in breast cancer.

Confronting Issues Related to Radiomics

Uniform Standard in Radiomics Studies

The most commonly mentioned consideration in all kinds of radiomic articles, whether about breast cancer or not, is always standardization as it relates to the deficiency in validation and reproduction in radiomic studies. From the included patients’ characteristics to the modeling and validation processes, there are many unstable variables in a collection of studies, among which, imaging-related processes may be the main ones. For example, although the studies we discussed above mostly involved DCE-MRI, device parameters including magnetic field intensity (3.0 T MRI can manifest higher resolution than 1.5 T) and other factors related to various manufacturers or differences in analyzed sequences may confer external implications in these studies. Additionally, it is apparent that modeling methods and even algorithm parameters are highly variable and are adopted distinctly in studies. All of these can lead to difficulties in evaluating and horizontally comparing the results of studies and applying radiomic models to clinical practice. Thus, in one way, as proposed by Lambin et al in a review in 2017, it is indispensable for researchers to disclose more detailed information about technical implementations and features analyzed. ¹³ On the other hand, uniform guidelines for specific radiomic implementation details such as the image biomarker standardization initiative, which is a cooperative project to standardize the extraction of image biomarkers from images, should be more extensively built and advanced.^16,62

In TIL-predicting radiomic studies, from the aspect of evaluating TILs, no gold-standard method has been established yet. As a result, the analytical validity of methods such as H&E staining, immunochemistry, or tissue microarrays cannot be assessed. This situation can hinder machine learning development related to evaluating TILs through digital pathology slides or radiological images. Additionally, specific thresholds for categorizing different TIL statuses have not been confirmed. Studies about TILs in breast cancer employed varying cutoff values according to different purposes (predicting prognosis, neoadjuvant chemotherapy pCR rate, etc) and different subpopulations. In addition, although detailed recommendations for standardizing and evaluating TILs were proposed by the International TILs Working Group 2014, whether most studies based on radiomic-TIL models adopted these recommendations uniformly is unclear.^62,63

Inadequate Open-Access Data

Another urgent requirement may be data-sharing. As the training of models can require massive patient data, the collection of which is challenging for a single group or institution, data-sharing is needed for a larger scale of studies. ¹⁴ The open-source databases for cancer images and related information such as TCIA and TCGA and databases for other various diseases including MedPix, Grand Challenges, and AMRG Cardiac Atlas can be useful in promoting the accuracy of models. More databases including both image information and corresponding clinical, pathological, and genomic information are needed.

Dynamic Changes in TME

Moreover, it has already been realized that the TME shows changes associated with the performance of immunotherapy. For example, exhausted T cells are observed after PD-1 inhibitor treatment, which can block the efficacy of subsequent immunotherapy.^64,65 Changes in the TME especially TILs can also be observed in radiotherapy, chemotherapy, or targeted therapy.^66,67 Thus, it may be important to supervise dynamic changes in the TME. Moreover, it is important to acknowledge that image features may vary over time after antitumor therapy; for example, Yu et al, ⁴⁰ the aforementioned study, compared radiomic features that predict preoperative axillary lymph node metastasis (ALNM) before and after neoadjuvant chemotherapy and found that some features were upregulated while some were downregulated. Nevertheless, most radiomic studies have not analyzed TME-predicting models in a consistent cohort over the conduct of immunotherapy or other antitumor therapy. While it is not easy to acquire data for such analysis, more related studies are expected.

Heterogeneity of Tumor Microenvironment

It should also be noted that the TME can be distinct among different tumor-metastatic sites even in the same patient. For example, the study led by Lee et al ⁶⁸ found that diverse immune cell abundance including neutrophils and macrophages varied in different metastatic sites in breast cancer. However, all radiomic models predicting the breast cancer TME exclude stage IV patients (meaning distant metastasis of cancer). Thus, most radiomic models only focus on tumors localized to the breast, while patients with distant metastasis breast cancer represent a considerably large group. In addition, this group comprises the major source of data for TIL-related immunotherapy models. Radiomic studies of the TME based on such cohorts may have more general importance in the future. ⁶⁹ Actually, both studies by Yu et al ⁴⁰ in 2020 and 2021 also built the model to predict preoperative ALNM using MRI images of both the ALN and tumor regions. In addition, AUCs of 0.90 and 0.87 were achieved in validation cohorts, respectively, which were even higher than those for the diagnosis by radiologists. Although it is based on patients with early-stage breast cancer, it may suggest that radiomics based on multisite cancer is feasible. ⁷⁰ Similar studies were also conducted based on other solid tumors such as lung cancer and gastric cancer.^71,72 In another study published in May 2023, Zhao et al ⁷³ tried building a radiomic model using image features derived from different tissues including visceral metastasis, bone metastasis or nodal metastasis to predict immune therapy in advanced breast cancer. The result showed that the radiomic model obtained a high accuracy with an AUC of 0.994 (95% CI: 0.988 to 1.000) in the training cohort, and 0.920 (95% CI: 0.824 to 1.000) in the validation cohort, respectively. This outcome suggests that combining several radiomic features can result in more potential value in radiomic models.

Complex Constitution of Tumor Microenvironment

From another perspective, it is hard to evaluate the holistic immune status in patients with data on only certain components such as TILs, other stromal cells, or some molecules in the microenvironment. This is also a challenge in immunotherapy research. Patients with high levels of PD-L1 benefit little or even acquire hyperprogression from immune checkpoint inhibitor therapy, which can be explained by several hypotheses such as activated regulatory T (Treg) cells, clustered macrophages or other cell- or molecule-related mechanism.^4,74,75 It explicitly shows the complexity of the human body, and thus, for radiomics, more detailed information should be integrated if it is to be applied in clinical practice for predicting efficacy.

Future and Prospects

Increasingly advanced technologies are being wielded in radiomics in the generation of AI; besides generally used machine learning methods, neural networks and deep learning are being used in TME radiomic models, which can detect more intrinsic image features and be used to establish models that are more accurately predictive.^69,76,77 Additionally, as most studies have adopted MRI to conduct radiomic modeling, molecular imaging modalities, especially PET and SPECT imaging, are reported to potentially have critical roles in assessing the TME. Compared to conventional methods, they can not only be used to assess the infiltration of immune cells but also their activation status and proliferation abilities and even the status of components such as cytokines and enzyme indolamine 2,3-dioxygenase more directly, which are related to immunological function.^4,78,79 Thus, depending on ever-advancing radiotracers, radiomics based on these molecular imaging modalities may be more promising in the era of precise medicine, especially involving the use of immunotherapy. ⁸⁰ In any case, it is expected that more standardized radiomics with more enriched data-sharing will all these studies to be conducted more easily, and more multicenter prospective trials will be initiated.

Abbreviations

TME

tumor microenvironment

TILs

tumor-infiltrating lymphocytes

MR

magnetic resonance

CT

computed tomography

PET-CT

positron emission tomography-computed tomography

DCE-MRI

dynamic contrast-enhanced magnetic resonance imaging

SPECT

Single-Photon Emission Computed Tomography

18F-FDG PET/CT

Positron emission tomography with 2-deoxy-2-[fluorine-18]fluoro-D-glucose integrated with computed tomography

FDG

fluorodeoxyglucose

CEUS

contrast-enhanced ultrasound

AUC

area under the curve

AI

artificial intelligence

ROIs

regions of interest

CNNs

convolutional neural network

FCN

fully convolutional network

PCA

principal component analysis

LASSO

least absolute shrinkage and selection operator

SVM

support vector machine

DBNs

deep belief nets

LOOCV

Leave-One-Out Cross-Validation

CV

Cross-Validation

WSI

whole-slide images

SUVmax

maximum standardized uptake value

RFS

recurrence-free survival

WTMM

Wavelet Transform Modulus Maxima

FDA

Food and Drug Administration

TNBC

triple-negative breast cancer

PD-1

programmed cell death protein 1

PD-L1

Programmed cell death 1 ligand 1

H&E

hematoxylin and eosin

IHC

immunohistochemistry

mIHC/IF

multiplex IHC/immunofluorescence

CD8+T cell

cluster of differentiation-8 T lymphocyte

HER2

Human Epidermal Growth Factor Receptor 2

NK cell

Natural killer cell

TCIA

The Cancer Imaging Archive

TCGA

The Cancer Genome Atlas

T1WI

T1 Weighted Image

T2WI

T2 Weighted Image

GLCM

gray-level co-occurrence matrix

GLRLM

gray-level run-length matrix

NGTDM

neighborhood gray-tone difference matrix

RM-wholeFC

radiomics model based on all features of whole tumor

RM-priFC

radiomics model based on all features of peri-tumor

ypCR

Pathological Complete Response after Neoadjuvant Therapy

BI-RADS

Breast Imaging-Reporting and Data System

ALNM

Axillary Lymph Node Metastasis

DEGs

Differentially expressed genes

KEGG

Kyoto Encyclopedia of Genes and Genomes

GO

Gene Ontology Analysis

RNA

Ribonucleic Acid

MHC

major histocompatibility complex

ILs

interleukins

CSFs

colony-stimulating factors

IFNs

interferons

HNSCC

head and neck squamous cell carcinoma

CD27

cluster of differentiation-27

TIME

tumor immune microenvironment

HR

Hazard Ratio

IBSI

image biomarker standardization initiative.