AI-link omnipotent pathological robot: Bridging medical meta-universe to real-world diagnosis and therapy

Key fidings

To address the challenges in pathological diagnosis, a pathological metaverse called the artificial intelligence (AI)-link omnipotent pathological robot (ALOPR) has recently been developed. ALOPR comes from the field of remote sensing, in which images from different sensors are analyzed in a wide spectral range. It is designed for high-resolution multispectral imaging and intelligent analysis of tumor slices with multiple biomarkers. Unlike the traditional digital pathological slice scanner, ALOPR integrates imaging, visualization, AI multimodal diagnosis, spatial omics analysis, data encryption, accurate quantification, and the tumor microenvironment. This integration, along with improvements in efficiency, accuracy, and flexibility, enables ALOPR (Figure 1) to be used in hospitals at multiple levels, including rural hospitals, county hospitals, community hospitals, and tertiary hospitals.

Figure 1.

The function and network of ALOPR in bridging medical metaverse and real-world diagnosis and treatment

ALOPR integrates imaging, visualization, AI multimodal diagnosis, spatial omics analysis, data encryption, accurate quantification, and the tumor microenvironment. This integration is along with improvements in efficiency, accuracy, and flexibility.

Introduction

Pathology has long been regarded as a "bridge discipline" between basic medicine and clinical medicine, and its position is extremely important. However, there is currently an extreme shortage of pathologists. Due to variations in hospital sizes, case volumes, and disease distributions, pathologists often have areas of limited knowledge depending on where they practice. The ability to communicate pathological findings, especially during rapid pathological diagnosis and difficult pathological consultations, and to share pathologists and pathology subspecialty expert resources through the internet has become an urgent need in the real-world practice.

On the other hand, with the rapid development of molecular pathology, targeted therapy, and multidisciplinary diagnosis and treatment (MDT), the requirements for pathologists to interpret immunohistochemistry (IHC) and fluorescence in situ hybridization (FISH) have gradually shifted from qualitative to quantitative assessment. It is now necessary not only to determine the source of the tumor but also to evaluate the tumor microenvironment. Interpretations of these results, however, are often inaccurate and are difficult to reproduce. Given these limitations, it is often difficult to meet these clinical demands for personalized treatment of patients.

Translational

In response to these needs and challenges, an ALOPR platform that integrates satellite digital image remote sensing technology, AI technology, blockchain technology, and three-dimensional evaluation technology has been developed.

ALOPR was developed based on a new scalable heterogeneous edge computing architecture (ZL202110421778.X). Its processing capacity is several to several hundred times higher than that of modern computers but at the same cost. It collaborates with multiple processors, such as CPU, GPU, and FPGA, and uses dynamic allocation of resources and tasks between different computing units to achieve load-balanced working conditions. To allocate computing resources more reasonably, pipeline and asynchronous parallel working modes are adopted.

Based on this new platform, multiple functions mentioned above can be achieved. The basic function is a powerful imaging analysis function. First, multispectral imaging of H&E, thinprep cytologic test (TCT), immunohistochemistry, FISH, multiple immunofluorescences, and slices with other sources can be achieved simultaneously. Second, there is strong processing power. ALOPR can process and calculate 15 million pixels per second. For a 15 × 15-mm tissue section magnified 200 times, it takes less than 30 s to analyze. Third, a dual target adaptive scanning control method (ZL202110158033.9) has been developed to achieve a “coarse to fine” browsing process, balancing imaging resolution and expanded time. The entire slide image (WSI) is first implemented at low magnification; then, the area of interest is observed at higher resolution according to user needs.

The second novelty of ALOPR is its ability to support multiple diagnostic scenarios. First, it can realize the remote rapid pathological diagnosis of intraoperative frozen sections. According to the actual measurement of the Chinese Food and Drug Administration, it takes only 2 min for the entire process of remote rapid consultation during surgery (including high-resolution imaging). Compared with the 11-min operation time on the platform mentioned in the references,^1,2 the efficiency of ALOPR has been improved by more than 200%. Second, it can support remote multidisciplinary therapy (R-MDT).

The third novelty of ALOPR is AI computational pathology. ALOPR integrates tumor detection and classification, image segmentation, cell detection and counting, mitosis detection, tumor grading, and AI-based tumor cell recognition models.^3,4,5 In addition, a high-resolution volume information fusion method with image depth of field (ZL2020, 115131625.2) was proposed to improve the recognition rate. A new registration method of FISH and H&E WSI was proposed to analyze the tumor microenvironment (ZL202011301331.0, ZL202111558334.7, ZL20211554677.1).

Specifically, when detecting multiple IHC and FISH indicators in consecutive tissue slices, the common cystic structures, blood vessels, and nerves in the tissue slices are compared to lakes, rivers, and mountains in remote sensing images. They are labeled with longitude and latitude like remote sensing satellite images and finely registered using shape geometry. Space localization and synchronous observation can be performed on the same cell or group of cells located in consecutive slices. In this way, IHC/FISH-positive cells can be reflected to the continuous digital slices, and their three-dimensional reconstruction can be conducted. The statistics of positive and negative cells can be obtained on those continuous digital slices with obvious cell contours. Furthermore, the expression of multiple different biomarkers (proteins or nucleic acids) at the same site can be compared and observed using pseudocolor skill. Their expression intensity can be objectively evaluated. Thus, in analysis of the tumor microenvironment, ALOPR can intuitively and accurately analyze the expression and spatial relationship of multiple biomarkers (proteins or nucleic acids) in different cells in the same region, help pathologists obtain accurate and quantitative analysis results, provide a basis for individualized treatment, such as immunotherapy and targeted therapy, and evaluate the prognosis of patients.

Applications

At present, the application and development of ALOPR is still in its early stages. In early trials of practical applications, through extensive data collection and deep learning, ALOPR established an AI-based transistor cell recognition model for cervical cancer, breast cancer, lung cancer, and lymphoma.^4,5 In the human-machine combined film reading mode, the sensitivity and specificity for cervical cytology interpretation reached 95% and 78%, respectively. However, these results are far from sufficient because in deep learning, the number and diversity of learning samples are very important, and the accuracy and efficiency of deep learning are determined by both model algorithms and hardware efficiency. Massive data and guidance from pathologists are keys to its continuous evolution. In addition, the uniqueness of ALOPR lies in that it not only pays attention to and learns pathomorphological characteristics but that it also focuses on the evaluation of the tumor microenvironment, including the spatial distribution and accurate quantification of protein and nucleic acids. ALOPR establishes an accurate digital diagnosis and treatment model for each patient, which requires the comprehensive analysis of AI interpretation results combined with the patient’s clinical data and survival curve in the next step. Only then can we ultimately provide a new theoretical basis and data support for precise digital diagnosis of pathology and precise digital treatment of clinical practice and build an AI intelligent diagnosis and clinical decision-making assistance system that serves the medical metaverse.

Perspectives

Currently, due to relatively limited data accumulation, the accuracy rate of ALOPR still has room for improvement. Its development will follow the accumulation of massive digital pathological data collected by various terminals and the continuous correction and guidance of many pathological experts. In the future, each ALOPR can work independently, and more importantly, they can be widely distributed to form an extensive network. Furthermore, it will provide an immersive experience based on extended reality technology and generate a mirror image of the real world based on digital twin technology. It will build an economic system based on blockchain technology and closely integrate the digital pathology of the virtual medical world with the morphological pathology of the real medical world in terms of spatial distribution, precise counting, and causal correlation. With widespread application and continuous iterations, ALOPR will become an AI assistant in real-world pathological diagnosis, a fulcrum of the medical metaverse, and a cornerstone of digital tumor therapy.

Patent number

The investigation related to this paper (ZL202110421778.X, ZL2020115131625.2, ZL202110158033.9, ZL202111558334.7, ZL202011554677.1, ZL202011301331.0) has authorized six invention patents in China between 2021 and 2022. The intelligent pathological stereometry analyzer received registration from the National Medical Products Administration of China on Nov 9, 2022 (registration certificate no. XZZ 20222222047).

Published Online: August 9, 2023