The Role of Geometry in Convolutional Neural Networks for Medical Imaging

Abstract

Convolutional neural networks (CNNs) have played an important role in medical imaging—from diagnostics to research to data integration. This has allowed clinicians to plan operations, diagnose patients earlier, and study rare diseases in more detail. However, data quality, quantity, and imbalance all pose challenges for CNN training and accuracy; in addition, training costs can be high when many types of CNNs are needed in a health care system. Topology and geometry provide tools to ameliorate these challenges for CNNs when they are integrated into the CNN architecture, particularly in the data preprocessing steps or convolution layers. This paper reviews the current integration of geometric tools within CNN architectures to reduce the burden of large training datasets and offset computational costs. This paper also identifies fertile areas for future research into the integration of geometric tools with CNNs.

Overview of Medical Imaging Needs

Article Highlights.

• •Convolutional neural networks (CNNs) in medical imaging face data imbalances and high training costs.
• •Topological data analysis and geometric approaches can enhance CNN performance.
• •Persistent homology improves the accuracy and robustness of CNNs.
• •Geometry in CNNs allows diverse input data and efficient feature extraction.
• •Collaboration between deep learning researchers and geometers could revolutionize personalized care.

Medical imaging plays a crucial role in modern health care by providing valuable visual information that aids diagnosis, treatment planning, and medical research. It consists of an array of noninvasive or minimally invasive techniques that enable clinicians to visualize internal structures and processes within the human body. The clinical needs for medical imaging are vast and diverse, encompassing various applications. One of the primary clinical needs for medical imaging is diagnostic imaging of patients. Multiple modalities, such as x-rays, ultrasonography, computed tomography, magnetic resonance imaging (MRI), and positron emission tomography (PET), are used to assess and diagnose different medical conditions. These imaging techniques help identify and localize abnormalities, detect tumors, evaluate the extent of injuries, and provide valuable information for guiding treatment decisions. In addition to diagnostic purposes, medical imaging plays a crucial role in medical research. It enables scientists and researchers to explore the human body in a noninvasive manner, facilitating the study of anatomical structures, (patho)physiological processes, and disease progression. Medical imaging techniques are extensively used in clinical trials and longitudinal studies to monitor the effectiveness of treatments, evaluate disease progression, and understand the underlying mechanisms of various conditions.1, 2, 3

Moreover, medical imaging aids in surgical planning and intervention. It provides surgeons with detailed preoperative images, allowing them to visualize anatomical structures, identify critical areas, and plan the safest and most effective surgical approach. In addition, real-time imaging during minimally invasive procedures, such as image-guided interventions, helps guide the placement of instruments and monitor the procedure’s progress. Furthermore, medical imaging has a vital role in monitoring treatment response. Serial imaging studies enable clinicians to assess the effectiveness of therapies, track changes in disease progression, and make necessary adjustments to treatment plans. This is particularly important in oncology, where imaging techniques like MRI and PET are used to evaluate tumor responses to chemotherapy, immunotherapy, and/or radiation therapy.^4,5

The advancements in medical imaging technology continue to expand its capabilities, allowing for more accurate and detailed imaging, ultimately improving patient care and outcomes (Figure 1).

Figure 1.

An example of medical imaging. The structure and internal components of a standard positron emission tomography (PET)/magnetic resonance imaging (MRI) scanner with the typical pathway for image acquisition. The color signifies uptake of fluorodeoxyglucose radiotracer. Red is the highest uptake, blue is the lowest.

Overview of Past Uses of Convolutional Neural Networks

Convolutional neural networks (CNNs) have revolutionized medical image analysis, providing radiologists with powerful tools for accurate detection and diagnosis. Convolutional neural networks natively use optimized topological mapping in their convolution layers to extract meaningful features from medical images in each layer to improve model fit to the outcome of interest.

One crucial aspect is the local connectivity pattern in CNNs. Convolutional neural networks leverage the concept of convolution, where filters slide over the input image to detect local patterns. This local connectivity is essential for radiologists because it enables the network to capture local features, such as edges, textures, and shapes, that are crucial in medical image analysis.⁶ For instance, in mammography, CNNs can identify microcalcifications or architectural distortions, aiding in the early detection of breast cancer. Another vital topological aspect is the receptive field. The receptive field refers to the region of the input image that influences the activation of a particular neuron. In CNNs, receptive fields increase with the depth of the network.⁶ Larger receptive fields allow the network to capture global contextual information, which is particularly useful in medical image analysis. Radiologists can benefit from this aspect, as CNNs can capture contextual relationships between anatomical structures, facilitating accurate localization and segmentation tasks. Furthermore, the hierarchical nature of CNNs is highly relevant to radiologists. Convolution layers in CNNs form a hierarchical representation of the input image. Low-level features, such as edges and corners, are captured in the initial layers, whereas high-level features, such as complex structures and textures, are learned in the deeper layers. This hierarchical representation enables CNNs to learn complex patterns in medical images, making them capable of differentiating between normal and abnormal findings. Radiologists can leverage this hierarchical aspect to aid in the detection of abnormalities and diseases, such as tumors or lesions.^6,7

The concept of parameter sharing is another important topological aspect of CNNs. Parameter sharing refers to the use of the same set of weights (filters) across different spatial locations of the input image. This aspect allows CNNs to extract local features regardless of their location, reducing the number of parameters and improving efficiency. For radiologists, parameter sharing is advantageous as it enables the network to learn generalizable features, making the CNN robust to variations in image acquisition and patient positioning. In addition, the pooling operations employed in CNNs contribute to their topological aspects. Pooling layers downsamples feature maps by summarizing local regions. This process reduces the spatial dimensions while retaining the most salient features, allowing the network to identify features regardless of their precise location in the image or image orientation. Clinicians can benefit from this aspect, as CNNs can accurately identify anatomical structures even with slight positional variations, improving the robustness of the analysis.

Despite the strong progress that has been made over the past several years, CNNs face several challenges with respect to medical imaging data, such as relatively small sample sizes, rotational variation (particularly in digital pathology, as histological sections mounted to slides can have varying orientations), and data natively living in nonlinear spaces (such as brain imaging data).⁸ Mathematical tools from geometry and topology can ameliorate these issues by incorporating global and local data features into either data preprocessing steps or the CNN algorithm itself. Recent results suggest this combination of CNNs with geometry and topology can solve important problems facing medical image analysis, and these approaches suggest other advances that might come from geometry-aware feature engineering as a step in the CNN pipelines.⁸

Alternative Approaches

Topological data analysis (TDA) is a mathematical framework that has gained popularity and is used in a variety of fields, including medical imaging. Use of TDA in radiology provides a novel technique to evaluate and interpret complex medical images, providing additional insights to radiologists.^9,10 The capacity of TDA to capture and evaluate the topological aspects of anatomical structures or lesions inside images is one of its primary advantages in medical imaging. Traditional image analysis approaches frequently rely on pixel intensity or texture analysis, which may not adequately represent the structures’ underlying geometric and topological features. On the contrary, TDA concentrates on structure, shape, and connectivity, allowing for a more comprehensive understanding of its properties.¹¹ Topological data analysis approaches can help radiologists obtain a better understanding of the spatial relationships and morphological features of anatomical structures or lesions. In addition, TDA-based techniques can detect voids, tunnels, or higher-dimensional structures in imaging data, allowing radiologists to better grasp the complex patterns and connections between distinct regions of interest and between different groups of individuals and patients. Topological data analysis can also help distinguish between different types of anatomical structures or lesions on the basis of their topological features. By developing topological representations of imaging data, radiologists can spot small alterations in the topology of structures that may be suggestive of pathological changes or illness progression.⁸ This has the potential to help them detect and classify numerous medical diseases more accurately. In addition, TDA can help with the integration of multimodal imaging data by merging information from various imaging modalities such as MRI, computed tomography, or PET scans. Using these tools, a more comprehensive and holistic understanding of the underlying anatomy and pathology can be obtained by examining topological features across several modalities, leading to more accurate diagnoses and treatment planning.

Topology in CNNs

The combination of TDA and CNNs has shown promise in improving CNNs for various applications in medical imaging. Elyasi and Moghadam¹² utilized TDA alongside the Xception neural network for the classification of skin lesions, achieving accurate identification and classification. The authors investigate the application of TDA in combination with the Xception neural network for the classification of skin lesions. They employ persistent homology to extract topological features from the images and combine them with the deep features learned by the Xception network. The experimental results report that incorporating TDA with the Xception network leads to improved classification accuracy for skin lesion recognition tasks. The integration of topological information helps capture the structural characteristics of skin lesions, enabling more accurate classification.¹²

Ghafuri et al⁶ incorporated topological concepts into CNN convolution layers, aiming to enhance accuracy and performance in medical image analysis. This study explores the utilization of topological aspects in CNN architectures for medical image analysis. They propose a novel approach that incorporates the topological properties of images through persistent homology, a technique from algebraic topology. The authors report the effectiveness of their approach by applying it to a dataset of medical images and comparing the results with traditional CNN models. They report that incorporating topological information can improve the performance of CNN models in terms of accuracy and robustness.⁶

Hajij et al⁷ proposed TDA-Net, a fusion of persistent homology and deep learning features, for COVID-19 detection from chest x-ray images, reporting improved accuracy. This introduces a method called TDA-Net, which combines persistent homology with deep learning features for COVID-19 detection using chest x-ray images. The authors leverage persistent homology to capture the topological structure of lung regions and extract discriminative features for classification. They found the effectiveness of their proposed method by achieving high accuracy in COVID-19 detection compared with traditional deep learning models. The integration of topological information through persistent homology enhances the model’s ability to identify patterns and irregularities in the lung regions, leading to improved detection accuracy.

These studies highlight the potential of TDA-CNN combinations in medical imaging tasks (Figure 2). However, previous research has reported the benefits of incorporating topological aspects through persistent homology into deep learning models for medical image analysis and skin lesion classification. The utilization of topological information enhances the models’ ability to capture complex patterns and irregularities in the data, leading to improved accuracy and robustness.

Figure 2.

Topological based persistent homology provides an accuracy boost to convolutional neural networks (CNNs) when added to the architecture that can potentially address deep learning limitations and illustrate the geometrical and texture information. The red box is the area that the topology or homology is being performed. MRI, magnetic resonance imaging.

Geometry in CNNs

Just as topology has contributed tools to improve CNN performance, so too have a few fields of geometry, and many of these applications involve the addition of geometric tools to pooling or convolution layers, much like the combination of TDA with CNNs. For instance, convolution layers can take curved image data (such as three-dimensional brain images) with locally flat geometries (called manifolds) as input and mapping spaces (Figure 3)¹³ or include rotation and translation operators on curved convolution mappings that manipulate parts of the original images as the algorithm learns good features and mappings¹⁴; these types of approaches have boosted CNN accuracy to state-of-the-art performances. The utilization of geometry in CNNs to improve accuracy involves the incorporation of geometric tools in pooling and convolution layers. In the case of curved image data, such as three-dimensional brain images, which possess locally flat geometries known as manifolds, geometric techniques can be employed. For instance, a convolution layer can be designed to include rotation and translation operators on curved convolution mappings, allowing manipulation of specific parts of the original images as the algorithm learns optimal features and mappings. These geometric approaches have been successful in enhancing CNN accuracy, leading to state-of-the-art performances in various applications.

Figure 3.

Varied geometry-based deep learning options for 2-dimensional and 3-dimensional space.

Geometric signatures and features can also be extracted from imaging data before fitting a CNN model to allow for more types of original input data or a combination of image data with other data types; matrices storing these features are then mapped across layers of the CNN.¹⁵ Features extracted from image data themselves can be engineered back into images before fitting the CNN; one example of this looks at self-similarity patterns (fractals) within sets of histological images to encode important geometric features defining structures natively rather than trust the algorithm to find those features itself, which can boost accuracy with class imbalance and increase training speed (Figure 4).¹⁶

Figure 4.

Convolutional neural network architectures commonly used in medical imaging. The workflow diagrams of the 2 methods. A, Conventional convolutional neural network processing of medical images. B, Geometric feature extraction followed by conventional convolutional neural network -based methods. CT, computed tomography.

Other approaches aim to extract features directly from convolution layers, project them into spaces with a different geometry, operate on data in this new space, and project it back to the convolution layers to train the model; this approach allows for feature rotation and translation in flat spaces rather than on the manifolds themselves; this simplifies the mathematics a bit and can improve accuracy when group imbalance exists.¹⁷ Although combinations of geometric operations with CNNs are a relatively new field, they perform well on some of the more challenging aspects of medical imaging, including group imbalance,^16,17 computational cost,¹⁶ and data type combination.¹⁵ Dan et al¹⁵ present a geometric-attention neural network for analyzing evolving manifold functional MRI data. The geometric-attention neural network incorporates geometric tools to capture the underlying dynamics of brain activity by modeling the evolving manifold structure. By considering the geometric aspects of the data, the geometric-attention neural network can effectively learn and represent brain dynamics, leading to improved analysis and understanding of functional MRI data. This approach reports the benefits of incorporating geometric techniques in neural networks for complex medical imaging tasks.¹⁵ In the study by Roberto et al,¹⁶ a novel ensemble model called the fractal neural network was proposed for histology image classification. The fractal neural network combines fractal geometry and CNNs to leverage the benefits of both approaches. Fractal geometry is employed to capture the intricate structural details in histology images, whereas CNNs extract high-level features for classification. The fusion of fractal geometry and CNNs in the fractal neural network results in improved classification accuracy for histology images. This highlights the potential of integrating geometric concepts, such as fractal geometry, with CNNs to enhance the analysis of complex image data.¹⁶ Liu et al¹⁷ propose a bundle geodesic convolutional neural network for the segmentation of diffusion-weighted imaging data in their paper titled “Bundle geodesic convolutional neural network for diffusion-weighted imaging segmentation.” The bundle geodesic CNN incorporates bundle geodesic curvature, a geometric property, into the CNN architecture to capture the spatial relationships and streamline orientations in diffusion-weighted imaging data. By considering the geometric information in the segmentation process, the bundle geodesic CNN achieves accurate and robust segmentation results. This reports the effectiveness of incorporating geometric properties, specifically bundle geodesic curvature, in CNNs for medical imaging tasks.¹⁷

Regarding the effect on group imbalance, the given papers do not explicitly discuss this aspect. However, by leveraging geometric tools and incorporating them into CNN architectures, it is possible to improve performance on imbalanced datasets. For example, techniques like oversampling or undersampling can be employed to balance the dataset during training, and geometric properties can aid in identifying relevant features and patterns, thus reducing the effect of group imbalance.

In terms of reducing computational cost, the incorporation of geometric techniques in CNNs can have varying effects. Although geometric-attention mechanisms and specialized architectures introduce additional computational complexity, they often lead to more efficient and effective processing of the data. For instance, by leveraging geometric properties, such as manifold structures or streamline orientations, CNNs can exploit the inherent structure of the data, leading to improved feature extraction and reduced redundant computations. In addition, geometric approaches can enable more focused and informative pooling or convolution operations, potentially reducing the overall computational cost. However, the specific effect on computational cost would depend on the complexity and implementation details of the geometric tools integrated into the CNN architecture.

Generalization of CNN by Exploiting Symmetries of the Medical Imaging

In the context of medical imaging, the generalization of CNNs for exploiting symmetries of data has important implications. Symmetries, which have long played a crucial role in understanding natural phenomena in various scientific fields, are now recognized as important considerations in constructing highly efficient neural networks, particularly when working with limited training data. For instance, if an input medical image contains a specific object and the object’s location is shifted within the same image, the features extracted by the CNN are expected to shift by the same distance. The convolution operation facilitates this process, ensuring the network remains sensitive to such spatial translations. However, generalizing the convolution theorem to encompass various other symmetries is critical to building efficient neural network architectures for medical imaging. In particular, techniques like Spherical CNN and Mesh CNN have proven highly valuable in scenarios where rotation and gauge symmetries are present.¹⁸ These approaches enable the network to effectively learn and exploit rotational and gauge symmetries, respectively, enhancing its ability to extract meaningful features from medical images. By leveraging the underlying symmetries present in medical imaging data, researchers and practitioners can develop more powerful and efficient CNN architectures. This, in turn, can lead to improved accuracy and performance in tasks such as image classification, segmentation, and detection within the medical imaging domain.¹⁸

Future Perspectives

Geometry and topology can combine with CNNs to create powerful medical image analytic pipelines, and recent advances incorporate only a few of the tools that geometry offers. Manifold learning and conformal mapping, in which high-dimensional imaging data can be mapped to a predetermined or best-fitting data space, may provide a good feature engineering step or pooling layer in a manner similar to persistent homology or translation/rotation operations. Smoothing tools, such as Ricci or Yamabe flow, may alleviate mounting artifacts that can distort histology data.^19,20 Decompositions of flow patterns into local and global patterns, as is done with the Hodge-Helmholtz decomposition, may simplify CNN classification of PET or echocardiogram data.²¹ The collaboration of medical deep learning researchers and geometers is necessary to solve some of the key problems in the field, such as accurate classification with small datasets or noisy data, and to develop a personalized approach to patient care.

In addition to the application of CNNs in medical imaging, transformer architectures, renowned for their success in natural language processing, can also be adapted and applied in this domain. The transformers’ ability to handle long-range dependencies could be advantageous in medical imaging, particularly in tasks involving sequential or time-series data, such as cardiac imaging or brain activity over time. They could be used to enhance manifold learning and conformal mapping, effectively processing high-dimensional imaging data.^22,23 Moreover, transformers could integrate with smoothing tools or aid in flow pattern decompositions, potentially improving classification accuracy and robustness. These implementations would further benefit from collaborations between deep learning researchers and geometers, leading to more personalized and efficient patient care.

Potential Competing Interests

Given his role as Editorial Board Member, Dr Bradley Erickson, had no involvement in the peer-review of this article and has no access to information regarding its peer-review. The remaining authors report no competing interests.