Abstract
Neural fields can efficiently encode three-dimensional (3D) scenes, providing a bridge between two-dimensional (2D) images and virtual reality. This method becomes a trendsetter in bringing the metaverse into vivo life. It has initially captured the attention of macroscopic biology, as demonstrated by computed tomography and magnetic resonance imaging, which provide a 3D field of view for diagnostic biological images. Meanwhile, it has also opened up new research opportunities in microscopic imaging, such as achieving clearer de novo protein structure reconstructions. Introducing this method to the field of biology is particularly significant, as it is refining the approach to studying biological images. However, many biologists have yet to fully appreciate the distinctive meaning of neural fields in transforming 2D images into 3D perspectives. This article discusses the application of neural fields in both microscopic and macroscopic biological images and their practical uses in biomedicine, highlighting the broad prospects of neural fields in the future biological metaverse. We stand at the threshold of an exciting new era, where the advancements in neural field technology herald the dawn of exploring the mysteries of life in innovative ways.
Introduction
Organisms inhabit a complex three-dimensional (3D) world, and the integration of advanced biotechnologies such as computed tomography (CT), magnetic resonance imaging (MRI), and microscopy techniques is revolutionizing our understanding of this domain. These technologies, while providing an abundance of two-dimensional (2D) biological images, pose a fundamental question: How can we transcend their inherent limitations to unlock a deeper understanding of the biological mysteries in 3D?
Neural field, powered by deep learning algorithms, utilizes signals encoded in its weights as implicit neural representations (INRs), enabling the transformation of discrete pixel coordinates into continuous space. This effectively converts 2D images into 3D spatial models without being limited by resolution and with precision. Thus, it plays a crucial role in shaping the metaverse in the realm of virtual reality (VR/AR), which is evidenced in the creation of virtual autonomous driving and virtual worlds.1,2 As this technology advances into the biomedical field, it will pave the way for the creation of a new biological metaverse, where neural field enables precise 3D reconstruction across the spectrum of biology, from macro-level structures, such as human anatomy,3 to micro-level entities, including biomolecules and cellular structures. This capability enables researchers to intuitively observe, analyze, and even manipulate these complex biological structures within a virtual environment. Not only does this greatly enhance our understanding of life sciences, but it also accelerates the pace of new drug discovery and the development of disease treatment strategies (Figure 1).
Figure 1.
Applications of neural fields in macro- and microbiology
Initially gaining attention in macro-biology, the neural field has since led researchers to explore its application in medical imaging techniques like CT and MRI. This exploration aims to overcome limitations in resolution, annotation, and image reconstruction. Traditional methods, crucial in scenarios in which enhancing image resolution is vital, often fall short due to their dependence on resolution and reliance on extensive datasets, making them less effective for super-resolution tasks. For instance, in practical medical imaging processes like CT, positron emission tomography, and MRI, prolonged scanning durations can result in minor patient movements, leading to motion artifacts that blur images or obscure structural details. The intrinsic characteristic of a neural field is its ability to generate a continuous field from spatial coordinates, which facilitates the reconstruction of obscured structures and enables super-resolution reconstruction. This highlights its exceptional capability to capture and reproduce complex spatial details. CuNeRF, 4 the Cube-based Neural Radiance Field, significantly contributes to this field by learning and generating higher-resolution images without the need for high-resolution data. This capability allows for image display at various resolutions and from different viewpoints, greatly enhancing the adaptability and flexibility of super-resolution processing in medical imaging. Overall, the neural field has significantly improved image resolution capabilities in macroscopic biological studies compared with traditional methods.
In microbiology, accurately reconstructing the spatial structures of proteins and other biomolecules through single-particle cryogenic electron microscopy (cryo-EM) is one method of understanding the processes of life. However, the heterogeneity of structures presents a significant challenge for the reconstruction of cryo-EM images. In the process of 3D structural reconstruction, the issue arises because each image may represent a different conformation of the molecule, rather than a single fixed structure. This means that reconstructing a unified 3D model from these images becomes more complicated using traditional methods, as it requires considering the potential structural variations of each molecular instance. CryoDRGN represents a significant breakthrough in cryo-EM data processing and 3D structural reconstruction. Unlike traditional methods that rely on discrete clustering to address structural heterogeneity and require manual intervention to determine molecular poses, CryoDRGN utilizes a neural field-based deep learning architecture to achieve ab initio unsupervised reconstruction of continuous distribution 3D volumes directly from unlabeled 2D images. This approach simplifies the 2D image modeling process, overcomes the challenge of structural heterogeneity, and captures the dynamic structural changes of biomolecules without the need for prior knowledge. Building upon the success of CryoDRGN, its successor, CryoDRGN2, further expands the capabilities of this technology. By incorporating improved latent variable models and global search strategies, CryoDRGN2 enhances efficiency and accuracy in handling the issue of unknown poses. These upgrades significantly boost its ability for ab initio reconstruction, especially in processing structurally heterogeneous datasets. Compared with previous technologies, both CryoDRGN and its evolved version, CryoDRGN2, not only demonstrate efficiency and accuracy in handling complex datasets but also open new pathways for studying protein dynamics and complex biomolecular mechanisms, offering a powerful new tool for research in the biomedical field.5,6
Key applications of neural fields in biological image processing
The integration of neural fields into biological image processing has led to groundbreaking advancements, notably facilitating the transition from 2D imaging to comprehensive 3D analysis. This progress covers critical tasks such as image registration, where aligning and integrating images from different sources becomes more precise; reconstruction, which now allows for the creation of detailed 3D models from sparse data; semantic segmentation, enhancing the ability to distinguish between different biological structures automatically; and data compression, significantly improving storage and transmission efficiency without compromising detail. These developments not only open new avenues for handling and interpreting complex biological data but also represent a paradigm shift in biomedical imaging. By providing a new perspective for comprehensive 3D studies, the neural field is setting new standards for accuracy and efficiency, pushing the boundaries of what is possible in the visualization and analysis of biological structures.
Biomedical images, such as MRI, often exhibit a multi-layered structure. Due to their imaging characteristics, these images present intervals between layers. In medical diagnostics, precise alignment and reconstruction of these images are crucial for identifying overlapping areas within a unified coordinate system. This alignment process requires learning a transformation function to align coordinates from source to target images. Conventional alignment methods usually involve complex steps and are based on fundamental assumptions about the mechanisms of transformation. Neural field offers a smooth, coherent approach to handling changes and distortions in images. NeSVoR7 effectively harnesses the consistency of the neural field combined with a continuous MRI slice acquisition model. This technique can be implemented at any resolution, making it particularly suitable for tasks like image registration, and can model transformation functions without relying on external assumptions. This synergy allows for the successful reconstruction of high-resolution 3D volumes from 2D slices that are affected by motion and artifacts. The challenge in CT reconstruction is sparse view, but the adoption of neural field is increasingly recognized as an effective strategy to address this. It proficiently translates coordinates into intensity values, thus significantly reducing errors within the projection space. This approach has the added advantage of obviating the need for extra pairs of projection and image data, streamlining the process. Implicit deformable image registration (IDIR)8 employs a neural field for spatial alignment, associating each coordinate in a reference image with a corresponding position in a dynamic image. Successfully tested on the DIR-LAB dataset for four-dimensional chest CT registration, IDIR has surpassed all existing deep learning-based approaches without the need for folding or dependence on specific training data, signifying a notable advancement. These technologies demonstrate exceptional capabilities in handling complex image registration and reconstruction tasks, significantly enhancing the accuracy and efficiency of the diagnostic process. The advancements in the neural field are not only crucial for technological development in the field of medicine but also open new possibilities for improving patient diagnosis and treatment. Single-cell spatial transcriptomics (scST) struggles to completely address discontinuities between layers, and biologists are highly interested in the behavior and transcriptional information of cells in a continuous state, making accurate alignment between layers crucial. The success stories in registration and reconstruction using the neural field in CT and MRI provide methodological guidance for more accurately predicting spatial expression of genes in single-cell spatial transcriptomics. We can apply the neural field to scST, leveraging its advantage in handling inter-layer discontinuities, to provide biologists with transcriptional information of cells in a continuous state and more precise spatial expression predictions.
Medical imaging faces significant challenges in accurately interpreting images with complex topologies, a task that becomes particularly daunting when dealing with fine structures in CT and early cancer detection in immunohistochemistry images. Traditional image processing methods often struggle due to their computational inefficiencies and limitations in handling such complexity. For example, identifying subtle structures within CT images is notoriously difficult, and extracting early cancer markers from immunohistochemistry images is equally challenging. The high level of precision required in medical diagnostics necessitates extensive expert involvement and demands careful consideration of patient privacy in data acquisition. Thus, efficiently obtaining accurate topological markings in medical imagery stands as a critical issue. In projects like Retinal INR,9 the incorporation of the neural field not only improves resolution in retinal vessel segmentation but also amplifies the details of capillary structures, critical for accurate diagnosis. The success of these technologies in analyzing macroscopic biomedical images offers valuable insights for their application in microscopic domains, like early cancer detection through immunohistochemistry image analysis. The integration of the neural field into medical imaging elevates the capability to discern finer details, heralding new possibilities in image analysis and early disease diagnosis. This showcases the profound impact of artificial intelligence in reshaping health care, illustrating the transformative potential of technology in medical diagnostics.
Efficient management of intricate biomedical data, spanning various tissue types and detailed imagery, presents a significant challenge. Conventional data compression often falls short for such complex data. Neural field, leveraging neural networks like multilayer perceptrons, marks an advancement in data representation. It efficiently encodes large volumes of data with fewer parameters while preserving essential details and adapting to unique data features. This efficiency is crucial for both storage and precise data reconstruction. Methods like NeRV10 exemplify the potential of neural field compression, showcasing a paradigm shift in biomedical data processing and analysis. As this technology continues to evolve, it holds promise for revolutionizing the interpretation of biomedical images, from macroscopic to microscopic levels.
Unveiling the future of biomedical research with neural fields
In summary, the development of neural fields represents not only a significant breakthrough in scientific innovation but also a profound revelation for the future direction of biomedical research. This technology, particularly in its transition from interpreting 2D biological images to a comprehensive understanding in 3D, has demonstrated its efficiency and intelligence. It provides more precise and powerful tools for medical research and diagnostics, paving new ways to explore the complex three-dimensionality of life sciences, building a biological metaverse. Neural field has the potential to fundamentally change our interpretation and application of biomedical images, offering unprecedented depth and breadth in unveiling the mysteries of the 3D biological world. This advancement is not just a reflection of technological progress but signifies a major leap in our understanding and manipulation of complex biological systems. It heralds a future where medical and biological sciences delve deeper into the 3D structures and functions of life.
Acknowledgments
Declaration of interests
The authors declare no competing interests.
Published Online: April 19, 2024
Contributor Information
Hao Zhao, Email: zhaohao@air.tsinghua.edu.cn.
Guangchuang Yu, Email: gcyu1@smu.edu.cn.
References
- 1.Cheng S., Zhang Y., Li X., et al. Roadmap toward the metaverse: An AI perspective. Innovation. 2022;3(5) doi: 10.1016/j.xinn.2022.100293. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Mildenhall B., Srinivasan P.P., Tancik M., et al. Nerf: Representing scenes as neural radiance fields for view synthesis. Commun. ACM. 2021;65(1):99–106. doi: 10.1145/3503250. [DOI] [Google Scholar]
- 3.Molaei A., Aminimehr A., Tavakoli A., et al. 2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW) IEEE Computer Society; 2023. Implicit Neural Representation in Medical Imaging: A Comparative Survey; pp. 2373–2383. [DOI] [Google Scholar]
- 4.Chen Z., Yang L., Lai J.-H., et al. 2023 IEEE/CVF International Conference on Computer Vision (ICCV) IEEE Computer Society; 2023. CuNeRF: Cube-Based Neural Radiance Field for Zero-Shot Medical Image Arbitrary-Scale Super Resolution; pp. 21128–21138. [DOI] [Google Scholar]
- 5.Zhong E.D., Lerer A., Davis J.H., et al. 2021 IEEE/CVF International Conference on Computer Vision (ICCV) IEEE Computer Society; 2021. CryoDRGN2: Ab initio neural reconstruction of 3D protein structures from real cryo-EM images; pp. 4046–4055. [DOI] [Google Scholar]
- 6.Zhong E.D., Bepler T., Berger B., et al. CryoDRGN: reconstruction of heterogeneous cryo-EM structures using neural networks. Nat. Methods. 2021;18(2):176–185. doi: 10.1038/s41592-020-01049-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Xu J., Moyer D., Gagoski B., et al. Nesvor: Implicit neural representation for slice-to-volume reconstruction in mri. IEEE Trans. Med. Imaging. 2023;42(6):1707–1719. doi: 10.1109/TMI.2023.3236216. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Wolterink J.M., Zwienenberg J.C., Brune C. Implicit Neural Representations for Deformable Image Registration. Proc. Mach. Learn. Res. 2022;172(1-17):1349–1359. [Google Scholar]
- 9.Gu J., Tian F., Oh I.-S. Retinal vessel segmentation based on self-distillation and implicit neural representation. Appl. Intell. 2023;53(12):15027–15044. doi: 10.1007/s10489-022-04252-2. [DOI] [Google Scholar]
- 10.Chen H., He B., Wang H., et al. Nerv: Neural representations for videos. Adv. Neural Inf. Process. Syst. 2021;34(2021):21557–21568. [Google Scholar]