Artificial Intelligence in Colorectal Cancer: From Patient Screening over Tailoring Treatment Decisions to Identification of Novel Biomarkers

Abstract

Background

Artificial intelligence (AI) is increasingly entering and transforming not only medical research but also clinical practice. In the last 10 years, new AI methods have enabled computers to perform visual tasks, reaching high performance and thereby potentially supporting and even outperforming human experts. This is in particular relevant for colorectal cancer (CRC), which is the 3rd most common cancer type in general, as along the CRC patient journey many complex visual tasks need to be performed: from endoscopy over imaging to histopathology; the screening, diagnosis, and treatment of CRC involve visual image analysis tasks.

Summary

In all these clinical areas, AI models have shown promising results by supporting physicians, improving accuracy, and providing new biological insights and biomarkers. By predicting prognostic and predictive biomarkers from routine images/slides, AI models could lead to an improved patient stratification for precision oncology approaches in the near future. Moreover, it is conceivable that AI models, in particular together with innovative techniques such as single-cell or spatial profiling, could help identify novel clinically as well as biologically meaningful biomarkers that could pave the way to new therapeutic approaches.

Key Messages

Here, we give a comprehensive overview of AI in colorectal cancer, describing and discussing these developments as well as the next steps which need to be taken to incorporate AI methods more broadly into the clinical care of CRC.

Introduction

Colorectal cancer is the third most common cancer with close to 2 million new cases every year globally, and the second most common cause of cancer death in the USA [1]. Strikingly, the incidence of early-onset CRC in younger patients is even rising [2].

Since the 2010s, two trends in CRC medicine have evolved simultaneously. On the one hand, our understanding of CRC as clinically and molecularly heterogeneous disease has been improved by large clinical trials, the establishment of distinct molecular subtypes [3], and personalized treatment strategies, especially immunotherapy [4–7]. CRC as a very multifaceted malignancy combined with many different tailored treatment options warrants a comprehensive workup of every patient [7]. On the other hand, the role and application of artificial intelligence (AI) in general and particularly in medicine and oncology has gained substantially in relevance, in particular with multiple FDA (US Food and Drug Administration) approvals for AI applications [8–11]. Especially AI and deep learning (DL) approaches, which are able to process a variety of large and complex medical imaging data, may be beneficial in supporting doctors in their everyday work [12–15] and may also be potentially helpful in carrying out additional analyses that were previously not possible [16]. Even though AI and DL are not synonymous, most recent AI studies in the field of biomedical research are based on DL approaches, as deep neural networks have become the predominant approach to process image and text data in medicine [11, 17].

During the course of every CRC patient journey, visual data modalities play a key role along the way: from initial diagnosis via endoscopic screening and pathological examination of biopsies, over assessing resection specimens to evaluating treatment responses and/or staging via imaging, such as ultrasound, computer tomography (CT), and magnetic resonance imaging (MRI). Considering that the clinical and molecular landscape of CRC is very diverse [3], and many different tailored treatment options are already or will soon be available in precision oncology approaches [7], a complex and integrative analysis of all these visual data enabled by AI approaches is urgently necessary. Therefore, AI approaches could soon help in the clinical workup of every CRC patient along different steps (endoscopy, imaging, pathology). Hence, this review aimed at comprehensively summarizing the potential of AI/DL techniques in CRC clinical practice, as well as in CRC research.

AI for Lower Gastrointestinal Endoscopy

Prevention and/or early detection of CRC via different screening tools is necessary to improve patient outcomes as especially early stages of CRC are associated with an extremely good survival. The cornerstone of CRC prevention and early detection in many countries is colonoscopy [18, 19] or stool testing. As screening colonoscopy has been shown to reduce all-cause mortality [20–22], routine colonoscopy is implemented in several healthcare systems around the world. Many different trials demonstrated a significant reduction of CRC incidence and mortality by colonoscopy [23–27]. In colonoscopy, recent technological advances such as virtual chromoendoscopy improved the identification of suspicious lesions [28]. Particularly, the removal of adenomatous polyps via colonoscopy plays a crucial role in preventing fatalities associated with CRC [29]. Nowadays, different AI algorithms, trained on endoscopy imaging data, have been developed and are even becoming ready for clinical implementation. Adenoma detection rate (ADR) is a pivotal quality metric for evaluating the colonoscopy procedures and it has been demonstrated that an increased ADR is associated with a reduction of postcolonoscopy/interval CRC [30, 31]. Unfortunately, the ADR is significantly influenced by the examiner and demonstrates considerable variability, with a range of 7.0%–53% [31]. In a recent meta-analysis, the adenoma miss rate was found to be 26% for adenomas, 9% for advanced adenomas, and 27% for serrated polyps [32]. Several prospective randomized studies have demonstrated that AI-based computer-assisted detection (CADe) enhanced the ADR when compared to a control group without using CADe (Table 1).

Table 1.

Recent prospective endoscopy studies investigating AI-based approaches (CADe)

Author	Year	Country	Patients, n	Endpoint	with CADe	w/o CADe	p value	Comment
Wang et al. [33]	2019	China	1,058	ADR	0.291	0.203	<0.001	RCT
Liu et al. [34]	2020	China	1,026	ADR	0.390	0.230	<0.001	RCT
Liu et al. [35]	2020	China	790	ADR	0.2901	0.2091	0.0090	RCT
Repici et al. [36]	2020	Italy	685	ADR	0.548	0.404	<0.001	Multicenter RCT
Su et al. [37]	2020	China	659	ADR	0.289	0.165	<0.001	RCT
Wang et al. [38]	2020	China	1,046	ADR	0.340	0.280	0.0300	RCT
Gong et al. [39]	2020	China	704	ADR	0.160	0.080	0.0010	RCT
Shaukat et al. [40]	2021	USA	83	ADR	0.542	0.406	0.0280	Historical control group
Quan et al. [41]	2022	USA	300	APC	1.350	1.070	0.0099	Historical control group
Ishiyama et al. [42]	2022	Japan	1,836	ADR	0.264	0.199	0.0010	Propensity score-matched prospective study
Koh et al. [43]	2023	Singapore	298	ADR	0.304	0.243	0.0200	Performance of AI-guided colonoscopies was compared to endoscopist’s baseline performance as “fictional” control arm
Nehme et al. [44]	2023	USA	840	APC	1.080	1.040	0.6500	Historical control group and retrospective analysis of a prospectively maintained patient database

This is a selection of recent representative trials and not a guaranteed fully comprehensive list.

AI, artificial intelligence; CAD, computer-aided detection; APC, adenomas per colonoscopy; ADR, adenoma detection rate; RCT, randomized controlled study; w/o, without.

In contrast to these results, in a recent RCT involving highly skilled colonoscopists, the utilization of a CADe did not improve ADR within cancer screening programs [45]. Furthermore another meta-analysis indicated no improvement of adenomas per colonoscopy [46]. Recently, Ladabaum et al. [47] came to a similar conclusion in what they call a “pragmatic implementation trial.” Consequently, the evidence supporting the real benefit of CADe remains unclear and inconclusive. The discrepancies observed in the use of CADe warrant further research to accurately assess the true impact of AI in colonoscopy. Still, such systems are potentially beneficial, especially in identifying small lesions (for details, see Table 1 and also compare Hann and Meining [48] and Wei et al. [46]).

Here, it should be considered that not every small polyp will eventually progress to CRC, and CRC rate does not seem to be associated to the occurrence of non-advanced adenomas [49]. The majority of polyps, known as diminutive polyps, measure between 1 and 5 mm and have a very low prevalence of progressing to advanced adenoma, with an exceedingly rare transition to CRC [50]. Nevertheless, the clear association of advanced adenomas and CRC risk is a very strong argument for the need of increasing detection rates by AI-based algorithms, especially as ADR shows a great interobserver variability [51], which could be reduced by the implementation of AI approaches.

Besides increasing detection rates, AI-based approaches may support the exact characterization of lesions/polyps, and therefore establishing a diagnosis. Consequently, AI models possess the capability to differentiate between neoplastic and non-neoplastic polyps with high accuracy in real time [52, 53]. This AI-based real-time differentiation of harmless lesions could lead to an efficient “resect-and-discard” strategy (if validated in further clinical trials), reducing the workload of pathologists (see section: AI for Pathology-Based Biomarker Development; shortage of pathologists globally). Here, it should be considered that the ground-truth based on histopathologic diagnosis of hyperplasic polyps versus serrated lesions may be also somewhat blurred as these diagnosis itself carry substantial degree of interobserver variability among pathologists [54], warranting a careful interpretation of such AI-based diagnosis [11].

Another important aspect of AI-based approaches in colonoscopy could be the improvement of quality parameters, again aiming at standardization and thereby reducing interobserver variability. Previous research has demonstrated that an automatic quality control system, designed to manage the timing of the withdrawal phase, oversee withdrawal stability, and assess the quality of bowel preparation, can significantly enhance colonoscopists’ performance during the withdrawal phase and notably increase the adenoma detection [37].

Radiologic Image Analysis with AI

Although initial diagnosis of CRC is mainly based on endoscopic findings and subsequent histopathologic analysis, imaging modalities, especially ultrasound, X-rays, and high-resolution MRI gain importance in the pre- and post-therapeutic evaluation of CRC patients [7, 55, 56]. For example, exact evaluation of tumor stage prior therapy initiation is crucial in rectal cancer as those patients with locally advanced rectal cancer usually are eligible for neoadjuvant chemoradiotherapy, and this is usually performed with rectal MRI.

Indeed, several studies have shown quantitative radiologic imaging analysis techniques using AI-based approaches for feature-extraction and analysis of high-throughput medical imaging data, often summarized as radiomics, are able to predict metastases [57] as well as survival outcomes [58] based on pre-therapeutic imaging [59]. In the context of neoadjuvant chemoradiotherapy in rectal cancer, it is still a pressing clinical need to identify those patients who benefit and those who do not: Only 15–27% of all rectal cancer patients reach pathological complete response (pCR) with no residual viable tumor cells left after neoadjuvant therapy [60]. Several studies have now shown that radiomics approaches can predict pCR with comparably high accuracy in patients with locally advanced rectal cancer based on MRI images (± clinicopathological features). For a comprehensive overview of studies evaluating AI-based prediction of pCR after neoadjuvant therapy in rectal cancer we refer to the review by Jia et al. [61]. Combining pretreatment biopsies as well as MRI in a so-called radiopathomics approach yields even better results in predicting treatment responses [62], proving the great potential of multimodal AI approaches. Adequately predicting response to neoadjuvant therapy will likely gain in importance in the near future with immunotherapeutic as well as conventional chemotherapeutic treatment regimens likely to be implemented in routine clinical use not only in rectal but also in colon cancer [63, 64]. Assessing treatment response from imaging data is not only in the neoadjuvant but also the metastatic setting of outmost importance. According to the RECIST (Response Evaluation Criteria in Solid Tumours) [65], therapy response is assessed by morphologic changes in tumor size. Emerging therapy options as targeted and immunotherapies with novel patterns of response (e.g., pseudoprogression, immunotherapy-related adverse events) pose a challenge to the assessment of treatment response. In this context, Lu et al. [66] could show DL models are beneficial in characterizing treatment-related morphological changes as early on-treatment response in metastatic CRC.

Besides conventional radiologic imaging, building DL technologies upon molecular imaging, for example, positron emission tomography (PET)/CT, is likely to provide novel insights and opportunities, as molecular imaging with specific tracers reflects changes in the tumor itself (such as fluorodeoxyglucose [FDG] as metabolic marker) or the tumor microenvironment (such as fibroblast activation protein inhibitor for stromal changes). In the context of CRC, Lv et al. [67] could already prove that radiomic analysis of ¹⁸F-FDG-PET/CT in combination with clinical characteristics can help in predicting prognosis. As high intratumoral fibroblast activation protein (FAP) expression on a immunohistochemical level has already been linked to a poorer prognosis of CRC patients [68], including fibroblast activation protein inhibitor-PET/CT data into the DL pipeline could potentially lead to an even better patient stratification.

AI for Pathology-Based Biomarker Development

The definite diagnosis of CRC is based, as for almost every other cancer type, on the histopathologic analysis of biopsy tissue and/or resection specimens. Histopathologic evaluation is performed visually by well-trained pathologists on glass slides during clinical routine, stained with hematoxylin and eosin (H&E). However, over the last 10 years, digitalization of histologic slides by slide scanners as so-called whole-slide images has enabled the large-scale development of AI-based techniques in CRC pathology. Yet, many of those AI-based techniques still lack clinical implementation in routine diagnostic pathology [69]. After histopathologic evaluation, CRC tissue can be further used for immunohistochemical or molecular testing of prognostic and predictive biomarkers, such as MSI (microsatellite instability), BRAF, or KRAS mutational status. AI-based techniques are likely to play an important role in all these different fields of routine CRC pathology (Fig. 1) and also have the potential to transform tissue-based CRC research (Fig. 2).

Fig. 1.

Landscape of AI applications in colorectal cancer pathology. AI applications can be useful for several scenarios in CRC pathology, namely, in rendering the diagnosis and assessing morphological features such as the assessment of TSR, TILs, and invasion front biomarkers like budding or DR. Moreover, AI tools can directly infer (some) molecular features, such as mutational, MMR/MSI status, and CMS subtyping, from H&E slides, without (or with less) further molecular wet laboratory testing needed. Additionally, AI tools can help assess clinical meaningful outcomes directly from histologic slides, such as survival and recurrence, therapy response, and metastatic potential, for example, lymph node metastasis. For all these different applications, immediate clinical use cases are conceivable. AI, artificial intelligence; TILs, tumor-infiltrating lymphocytes; DR, desmoplastic reaction; MMR, mismatch repair (p, proficient; d, deficient); MSI, microsatellite instable; MSS, microsatellite stable; CMS, consensus molecular subtype; H&E, hematoxylin and eosin. Histologic H&E slides were obtained from https://portal.gdc.cancer.gov/ (TCGA-COAD, TCGA-READ) [70–72]. Created with BioRender.com.

Fig. 2.

Integration of AI approaches into CRC research as part of a comprehensive multimodal research pipeline. DL algorithms can help identify important morphological features with regards to different questions, such as therapy response, survival, metastatic potential, or molecular alterations, even on conventional routine H&E-stained slides. These histologic features can then be interrogated with novel molecular techniques that allow a high spatial or single-cell resolution (scRNA-seq and spatial transcriptomics). This will eventually lead to a better understanding of the underlying tumor biology, which can then on the one hand help to refine DL algorithms, and on the other hand potentially lead to novel, so far unknown biomarkers and/or therapeutic targets. Therefore, we believe an integration of AI approaches into a comprehensive multimodal research pipeline, that consists of conventional histopathology, molecular pathology, spatial and single-cell biology, as well as DL approaches, is beneficial and will likely foster biomarker and drug development in the near future. AI, artificial intelligence; DL, deep learning; CRC, colorectal cancer; H&E, hematoxylin and eosin; scRNA-seq, single-cell RNA sequencing; WSI, whole-slide image. Histologic H&E slide was obtained from https://portal.gdc.cancer.gov/ (TCGA-COAD:TCGA-AA-3552) [70–72] created with BioRender.com.

Diagnosis and Morphological Features

Based on H&E-stained tissue slides, pathologists not only diagnose primary or metastatic (lymph node or distant metastases) CRC but also assess several histologic features to assess the tumor biology, for example, budding, tumor-stroma ratio (TSR) or desmoplastic reaction (DR), and guide treatment decisions. With a global shortage of pathologists [73], that will become even more evident within the next years, AI-based tools could become crucial to facilitate and decrease the workload of pathologists to face this urgent problem. Several studies have shown that AI-based models can convincingly differentiate between normal tissue and CRC, which was comprehensively summarized by Davri et al. [74]. The most reasonable use case for AI-based tools to relieve pressure from pathologists would be ruling out typical biopsies, i.e., non-neoplastic and non-inflammatory, which goes far beyond only identifying neoplastic epithelium on the biopsies as also inflammatory conditions such as inflammatory bowel disease, acute inflammation, or collagenous colitis need to be detected by the model. Bilal et al. [75] could show by developing a weakly supervised DL model, called CAIMAN (Colorectal AI Model for Abnormality Detection), which reached a sensitivity of 0.99 in distinguishing between neoplastic, inflammatory, and atypical biopsies according to their definition, that this is indeed feasible if a large-scale validation as well as a subsequent clinical trial build upon these promising findings. Another time-consuming, repetitive task in CRC pathology is microscopical screening of lymph node metastases, especially when considering that lymph node harvest has been increased in the past by optimized surgical as well as pathological techniques [76, 77]. Recently, Khan et al. [78] could establish an AI-assisted diagnostic solution which seems to possess an almost perfect ability in identifying CRC lymph node metastases. For gastric cancer, it has already been shown that AI-assisted workflows improve detection of micrometastases and isolated tumor cells, and reduce the review time of histopathological lymph node assessment [79] – results which are probably equally true for CRC.

Beyond just diagnosing and detecting CRC, pathologists evaluate different histopathological features in CRC biopsies or specimens that are relevant for prognosis and/or therapy response. Several AI-based approaches could reach good, clinical-grade performance in all of these tasks, namely, in the evaluation of histologic subtypes, grade of differentiation, tumor budding, TSR, DR, pattern, tumor-infiltrating [80], and/or peritumoral lymphocytes (tumor-infiltrating lymphocytes [TILs]).

As the visual assessment of many of these features shows a considerable interobserver variability, AI-based approaches could help make these assessments more reproducible and more reliable. Even though low TSR, low TILs, and immature DR have shown to be prognostic and potentially even predictive biomarkers [80–85], none of these morphological biomarkers has been implemented on large scale in clinical routine. DL-based approaches, for example, for automated TSR [86, 87] or TIL quantification [88, 89] or reliable assessment of DR patterns [90], can potentially accelerate the clinical implementation of such biomarkers. Moreover, a quantitative DL-based biomarker assessment beyond just qualitative visual perceptions could ultimately lead to an even more accurate patient stratification and could realize the promise that computational and digital pathology foster the transformation of pathology in a more quantitative specialty.

Assessment of Molecular Alterations with AI

Besides these morphologically important features that reflect the tumor biology and can be reliably assessed by human pathologists, H&E slides carry even more information as the phenotype displayed in the H&E morphology can be considered as the ultimate reflection of the cancer transcriptional state and/or genotype. Exactly this knowledge that the genotype is expressed in the phenotype can be exploited by DL approaches predicting genetic alterations directly from H&E histology. This is appealing as H&E slides are available for every cancer patient, and normally genetic information is retrieved by additional time- and cost-consuming testing, which is only possible in certain areas globally. Three levels of molecular information can be distinguished, which all have been shown to be predictable directly from H&E histology without further molecular testing by DL approaches.

• 1.Mutational status of single genes [91–93], as BRAF, KRAS, and so on
• 2.Composite biomarkers displaying genetic signatures [91–98] (microsatellite status, molecular pathways, tumor mutational burden/homologous recombination deficiency)
• 3.Gene expression, especially consensus molecular subtypes (CMS) [99, 100] or pathway-derived subtypes (PDS) [101]

Even though DL-based assessment of single gene mutations such as BRAF or RAS mutations, which become relevant in the metastatic CRC setting, reaches high performance in some cases, with dramatically reduced costs of conventional sequencing methods, it is unlikely that DL-based approaches with moderate areas under the curve (as performance metric for DL-based models) will replace sequencing in the clinic in the very near future [102]. However, DL-based approaches could here become relevant for giving insights into tumor heterogeneity. Compared to this outlook, it is very likely that DL-based approaches will play an important role in MSI testing. MSI testing is not only performed to identify cases of Lynch syndrome but also in the context of immunotherapy, which is approved in the metastatic setting for MSI/dMMR (microsatellite instable/deficient mismatch repair) CRCs [4] and will probably soon become a routine neoadjuvant strategy in CRC [64, 103, 104]. Ruling out MSS cases could dramatically reduce the cases that need to be further investigated by immunohistochemistry – thereby reducing costs, workload and turnaround time. It has been shown in numerous studies that MSI status can reliably be assessed by DL algorithms not only on resection specimens but also on biopsies, which has eventually led to the approval of a first commercial DL algorithm for MSI detection from H&E images for routine clinical use in Europe in 2022 (MSIntuit, Owkin, Paris/New York). This platform has previously been validated by Owkin itself [94] and should now be validated by independent research groups and clinicians.

CMS, based on gene expression profiling, have been established for CRC several years ago aiming to better understand the underlying tumor biology. Even though the prognostic relevance of CMS as well as its potential predictive role have been demonstrated several times [3, 105, 106], widespread clinical implementation is lacking as additional testing in form of challenging assays is required for the establishment of CRC CMS. Sirinukunwattana could convincingly demonstrate that DL-based assessment of so-called image-based CMS (imCMS) is indeed feasible, enabling a biological stratification of CRC without performing wet laboratory testing. Once again, DL-based imCMS assessment could reveal spatial intratumoral heterogeneity – an issue that has so far often been neglected in treatment approaches and is potentially responsible for treatment failure/resistance in some cases [100]. In a subsequent study, these imCMS-based classifications now were predictive of neoadjuvant therapy in rectal cancer, which could be an important clinical use case, considering the low amount of available tumor tissue for further testing in biopsy material [99]. Just recently, in a similar approach, it could be shown that also CRC subtypes, established on a pathway level (pathway-derived subtypes 1–3, PDS1-3), can be predicted partially by DL algorithms directly from H&E-stained tissue slides. However, no PDS3-specific morphologic patterns could be observed in this study, indicating that not all changes on gene expression level necessarily lead to distinct visual features detectable on H&E [101]. Consistently with these findings, it has been shown for multiple cancer types, including CRC, that predicting gene expression from whole-slide images is feasible [107].

Prediction of Survival, Lymph Node Metastasis, and Therapy Response

Beyond evaluating and predicting known relevant features, DL models can also be trained to directly infer relevant outcomes, such as survival [108–110] recurrence, metastasis [111], and therapy response from H&E images. Recently, Foersch et al. could demonstrate that adding immunohistochemical stains highlighting the immune infiltrate into the DL pipeline leads to an even better patient stratification and has also predictive abilities in terms of neoadjuvant therapy in rectal cancer [112]. Hence, DL models can really help to predict treatment responses based on H&E histology and/or immunohistochemistry, especially in the neoadjuvant setting, which also has been shown in other studies [99, 113]. With the before-mentioned growing relevance of neoadjuvant immunotherapeutic as well as chemotherapeutic approaches not only in rectal but also in colon cancer, advances in AI are now opening up completely new alternatives for translational research in clinical trials like FOxTROT [63, 114]. For example, AI algorithms can be trained based on the comparison between pre-therapeutic biopsy and then regression assessment in the resected specimen to identify those patients who will particularly benefit from such a therapy. In the subsequent studies, both histologic evaluations (biopsy, resection specimen) as well as imaging modalities and molecular findings could be merged within multimodal AI applications for optimized treatment strategies.

Identification of Novel Biomarkers: Tumor-Adipocyte Interaction

One promise of AI approaches in medicine is to enable the discovery of completely new biologic mechanisms, and hence biomarkers and therapy targets. For CRC pathology, such an intriguing example exists Wulczyn et al. [110] could identify by deploying DL algorithms that so-called tumor adipose features (TAF) are relevant for the prognosis of CRC patients. These TAF could be reproduced in a pathologist validation study as independent new histopathologic biomarker [115]. Interestingly, Foersch et al. [112] could also highlight tumor-adipocyte interaction as so far underappreciated morphological feature in their multistain DL-based model (see in their paper in Fig. 5G). Jiang et al. [109] came to similar findings in their published multicentric end-to-end prognostication study, which highlights yet again the importance of tumor-infiltrated fat for a poor prognosis (see in their paper in online suppl. Fig. 14). Another DL model independently confirmed the important role of tumor cells adjacent to adipose tissue with regards to lymph node metastasis [111]. Moreover, deploying DL algorithms on early CRC H&E slides enabled the identification of inflamed adipose tissue as risk factor for lymph node metastasis [116]. Simultaneously, we established SARIFA (Stroma AReactive Invasion Front Areas) as novel histopathologic solely H&E-based biomarker in gastric cancer and CRC, defined by direct interaction between tumor cells and adipocytes at the invasion front, and could link SARIFA to a distinct tumor biology with an upregulation of lipid metabolism and immune dysregulation [117–121]. SARIFA as human-observed biomarker and TAF as DL-based developed feature both show striking similarities but also some subtle differences. Nevertheless, both concepts highlight the important and so far underappreciated role of tumor-adipocyte interaction in CRC pathology, which seems to be associated with an underlying biology. This distinct tumor biology is likely to have an influence on therapy response and may also be addressed therapeutically, by immunotherapeutic approaches or even with novel drugs specifically targeting lipid metabolism [122–124].

This example shows how the interaction of pathologists and DL models can help in the identification of novel biologically and clinically meaningful biomarkers. We believe that this implementation of DL-based approaches into the research pipeline could be extremely beneficial. Considering all the novel spatial and high-resolving molecular techniques, which are now available and already have been applied successfully on CRC specimens [125–130], such as spatial transcriptomics, multiplexing or single-cell RNA sequencing (scRNA-seq), DL algorithms could help identify new and so far underappreciated H&E features (e.g., visualized as attention maps/heatmaps), that could then be comprehensively studied with these novel techniques. The differentially expressed genes and/or dysregulated cell types could subsequently be exploited as biomarkers and/or targeted therapeutically. Of course, the combination of spatial/scRNA transcriptomics data and DL algorithms opens multiple other opportunities, such as prediction of gene expression that is initially not based on bulk but on spatial and/or scRNA-seq data [131, 132].

Critical Steps for Clinical Implementation

Many of the AI-based approaches presented in this review are likely to improve patients’ outcome or save time and reduce workload in already strained healthcare systems worldwide. However, there are some key obstacles that are necessary to overcome, namely, reimbursement, high investment costs, technical implementation (in particular computing and storage capacity), prospective randomized trials and real-world efficacy, willingness of patients and physicians to use and trust AI systems [69].

Without a clear reimbursement strategy, laboratories and hospitals have no incentive at all to integrate AI-based solutions into their workflow. In the case of colorectal cancer, for example, if only MSI/dMMR testing via immunohistochemistry, PCR-based approaches, or next-generation sequencing is remunerated, why should pathologists use solutions like MSIntuit [94] for ruling out pMMR/MSS cases prior molecular testing via DL-based predictions on H&E slides? Apart from that, technical implementation of all these solutions should be straightforward and best bundled together. If for every cancer type (such as CRC) and every test (e.g., testing for MSI/dMMR, prognosis prediction, therapy response prediction, etc.), the implementation of a different, separate DL solution into the in-house software environments is necessary, then the practical-logistical implementation is, apart from the question of cost, not feasible at all. After all, there is also considerable maintenance work for all the software/apps required. Moreover, just as in other parts of evidence-based medicine, high-quality clinical trials, if possible randomized (RCT) and prospective, are necessary to really prove that the use of DL-based algorithms do show a real benefit [133]. Besides the trial setting, this benefit must be reliably translatable into a real-world setting. Lastly, patients as well as physicians need to trust DL-based models. We firmly believe that this trust will grow with more and more evidence (RCT!) and explainability approaches (biological plausibility!) coming up.

Conclusion

The CRC patient journey involves many different steps that rely on a high-quality assessment of visual data: from initial diagnosis (endoscopy, histopathology) over staging (imaging, histopathology) to tailoring treatment decisions (histopathology, imaging). AI approaches have shown to be beneficial in all of these aspects and are now getting ready for clinical implementation. We believe it is likely that many new clinically approved AI-based support systems as well as biomarkers will be integrated into the clinical management of CRC patients. However, for clinical implementation, randomized clinical trials are urgently needed to validate all these approaches, mainly at improving the two key endpoints in cancer medicine: overall survival and quality of life. First prospective randomized clinical trials for AI-based tools are already under way. Of course, reducing costs and workload and enabling targeted therapeutic approaches, in settings where molecular testing is not available, could be also important reasons for the implementation of AI-based tools in the diagnosis and treatment of CRC patients. Moreover, the combination of high-resolution molecular techniques such as spatial transcriptomics or scRNA-Seq (single-cell RNA sequencing) with AI-based approaches could help in better understanding CRC biology and eventually lead to novel prognostic and predictive biomarkers as well as therapeutic targets.

Acknowledgments

We thank all researchers, whose scientific findings and efforts form the basis for this review article.

Conflict of Interest Statement

J.N.K. reports consulting services for Owkin, France (producer of MSIntuit), Panakeia, UK, and DoMore Diagnostics, Norway, and has received honoraria for lectures by M.S.D., Eisai, and Fresenius, not related to this study. B.M. has received compensation for travel expenses and fees for advisory board activities by AstraZeneca, Boehringer Ingelheim, Merck, MSD, BMS, Bayer, and Novartis, not related to this study. The other authors have no conflict of interest to declare.

Funding Sources

J.N.K. is supported by the German Federal Ministry of Health (DEEP LIVER, ZMVI1-2520DAT111) and the Max-Eder-Program of the German Cancer Aid (grant #70113864), the German Federal Ministry of Education and Research (PEARL, 01KD2104C), and the German Academic Exchange Service (SECAI, 57616814). This research was funded/supported by the National Institute for Health and Care Research (NIHR, NIHR213331), Leeds Biomedical Research Center. The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR, or the Department of Health and Social Care. N.G.R. is supported by the Manfred-Stolte Foundation (gastrointestinal pathology research).

Author Contributions

N.G.R. and J.N.K. conceptualized the article. N.G.R. drafted the article. J.S.E., K.V.T., and B.M. contributed to the article with their expertise and reflective improvements. All authors critically revised the manuscript and approved the final version for submission.

Funding Statement

J.N.K. is supported by the German Federal Ministry of Health (DEEP LIVER, ZMVI1-2520DAT111) and the Max-Eder-Program of the German Cancer Aid (grant #70113864), the German Federal Ministry of Education and Research (PEARL, 01KD2104C), and the German Academic Exchange Service (SECAI, 57616814). This research was funded/supported by the National Institute for Health and Care Research (NIHR, NIHR213331), Leeds Biomedical Research Center. The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR, or the Department of Health and Social Care. N.G.R. is supported by the Manfred-Stolte Foundation (gastrointestinal pathology research).