The metaverse in medicine

Abstract

The metaverse is an alternative digital world, accessed by means of dedicated audiovisual devices. In this parallel world, various forms of artificial intelligence meet, including individuals in the form of digital copies of real people (avatars), able to interact socially. Metaverse in medicine may be used in many different ways. The possibility to perform surgery at a distance of thousands of miles separating the patient from the surgeon, who could have also the possibility to visualize in real-time patient’s clinical data, including diagnostic images, obviously is very appealing. It would be also possible to perform medical treatments and to adopt pharmacological protocols on human avatars clinically similar to the patients, thus observing treatment effects in advance and significantly reducing the clinical trials duration. Metaverse may reveal an exceptional educational tool, offering the possibility of interactive digital lessons, allowing to dissect and to study an anatomical apparatus in detail, to navigate within it, not only to study, but also to see the evolution of the pathological process, and to simulate in advance surgical or medical procedures on virtual patients. However, while artificial intelligence is now an established reality in the clinical practice, the metaverse is still in its initial stages, and to figure out its potential usefulness and reliability, further developments are expected.

Introduction

In 1992, for the first time, Neal Stephenson in the novel ‘Snow Crash’ introduced the term ‘metaverse’, meaning with it a virtual reality (VR) going beyond the physical reality, that is an alternative virtual world in which individuals can create activities as they like and share them with other users.¹ The metaverse therefore appears as a parallel 3D reality located in the network, which can be accessed through dedicated technological tools such as glasses or audiovisual devices. It is characterized by a shared collective virtual space, based on the social interaction and created by the convergence of physical reality and virtually augmented digital reality. Real individuals can express their personalities and habits in the metaverse, using a digital alternative identity projected into an environment and context created by the convergence of the virtual world and the real world.^1–3

The metaverse is based on four technologies: VR, augmented reality (AR), mixed reality (MR), and extended reality (XR). Virtual reality is a technology service that allows users to experience a real-life environment in a virtual world created by digital devices; AR is a technological service that provides an environment in which a virtual object represented in 2D or 3D interacts with a real space; MR is a technology service that combines information from both the real and the virtual world to create a virtual space where the two worlds are combined; XR is a technology service that implements a concept that includes VR, AR, and MR, as well as another form of reality that will appear in the future. Extended reality is a generic technology that implements the metaverse and is therefore considered the main key to convergence with the industry (and with the healthcare industry in particular) to build a new social and industrial ecosystem.^2–4

From artificial intelligence to the metaverse

In order to understand the concept of the metaverse and how it differs from previous forms of technology, it is essential to consider the role that the ‘internet of things’ (IoT) plays in it.^2,4–6 The concept of IoT consists in the extension of the internet to the world of concrete objects and places, which acquire their own autonomy and digital identity in order to be able to communicate with other objects on the network and thus be able to provide services to users. The objects (the ‘things’) thus become recognizable and acquire intelligence thanks to the fact of being able, on the one hand, to communicate data about themselves and, on the other, to access aggregate information from other objects. Thus, based on the information received and/or sent via the network, the alarm clock will sound earlier in the event of rain or traffic problems, the accessories worn by the athletes will be able to transmit times and performances, speed, and distance travelled, to compete in time with other athletes in different areas of the world, or the intelligent medicine containers will be able to warn the patient or family members if they forget to take the medicine. All objects can acquire an active role, thanks to the connection to the network, and participate in the life of the individual, hence the introduction of the so-called ‘smart’ objects, which are characterized by certain properties such as identification, connection, localization, the ability to process data, and the ability to interact with the external environment. For decades, the use of tools and appliances has supported people in their daily lives (mobile phones and computers). The difference is that now, these tools are ‘smart’, that is they have a certain degree of autonomy of functions and interconnection with reality.

The goal of the IoT is to make the electronic world draw a map of the real one, giving an electronic identity to things and places in the physical environment. Objects and places equipped with radio frequency identification (RFID) labels or QR codes communicate information on the web or to mobile devices such as Personal Digital Assistants, tablets, or mobile phones. In other words, the IoT represents the bridge between the real world and the metaverse, or the tool with which to create the metaverse and from which to find the information necessary for its continuous updating.

The metaverse is therefore configured as a digital world still partially unexplored or in any case to be composed progressively, where people will be able to navigate and move through reproductions, also generated in their likeness (avatars and digital twins), more or less faithful graphic representations of users, able to interact in real time with other digital people, each with their own individuality. A dimension therefore where VR, AR, artificial intelligence, IoT, and various other technologies meet and complement each other to articulate physical and digital worlds.

The metaverse and medicine

Among the fields of application of the metaverse, medicine is a field of great relevance, and the term medical IoT (MIoT) represents the artificial model applied to medicine, which is facilitated by particular equipment such as AR and VR glasses.^2,4–7

The recent coronavirus disease 2019 (COVID-19) pandemic has greatly favoured the diffusion and technological development of increasingly sophisticated ‘remote’ telemedicine. Due to the need to maintain social distancing, people were forced to switch from the direct doctor–patient relationship to an electronic relationship based mostly on videoconferences. This solution can certainly be optimal for those consultations that do not strictly require a physical examination in the presence; however, the lack of physical contact between doctor and patient can reduce the empathy that commonly develops in a traditional medical examination.^8,9

The metaverse can partially overcome this barrier in the context of a 3D reality in which visual and auditory information is combined with tactile sensations thanks to particular devices such as gloves with appropriate sensors. Furthermore, thanks to AR, the physician simply through his smart glasses would be able to visualize, within his field of vision, all the patient’s clinical and instrumental data, diagnostic images, and complete electronic records, with no need to get such information from the computer. Furthermore, the system will be able to highlight errors during the diagnostic–therapeutic process or, thanks to specific smart devices (mobile phone, smartwatch, and vital parameters detectors), it would be able to signal the possible inadequate adherence of the patient to the assigned therapy.

In various contexts, MIoT has proven effective in serving the patient community. A case in point is the Asthma Prevention Application (AHA), developed in the USA, which was designed to conduct healthcare research on a large scale and provide real-time monitoring of air pollution. Based on the analysis of patients’ electronic asthma diary data, combined with atmospheric data, this application can predict acute asthma attacks, contributing to primary and secondary prevention of the disease.¹⁰

Another example of the MIoT application is provided by the Japanese Toshiba, which has developed an artificial intelligence device made up of wrist sensors and a palmtop, able to analyse and monitor the user’s health, activities, and personal habits on a daily basis, providing reminders and advice for an appropriate healthy diet and regular exercise, tailored to the specific individual. Based on the characteristics of the arterial pulse, movement, heart rate, and electro-dermal activity, artificial intelligence was found to play a key role in making behavioural changes and in reducing the risk of lifestyle diseases. The software achieved 90% accuracy in detecting user activities such as eating and exercising.¹¹

The taking of digital care paths to extremes as a physician–patient interaction tool led to create virtual hospitals in the metaverse, such as the Hospital Alfa in the virtual city called Aimedis Health City, where doctors and patients of different origins and nationalities meet and interact, sharing their experience and knowledge.

Currently, in the ‘real world’, efforts of the health authorities are aiming at concentrating resources (i.e. technologies and skills) in reference and highly specialized centres. This centralization of care can generate difficulties for patients (especially the elderly and frails with little autonomy) who live in peripheral areas and are geographically distant from the reference centre for excellence of medical care. In this context, the MIoT can be a valid support to the doctor’s activity, through its three fundamental characteristics: complete perception, reliable transmission, and intelligent processing.

The possibility of a VR integrated by intelligent medical devices capable of reconstructing a faithful avatar or ‘digital twin’ of the patient (a twin virtual copy of the patient), supported also by current diagnostic tools (including computed tomography and positron emission tomography), may at least partially compensate the lack of a direct doctor–patient relationship.

Another application of the metaverse in medicine was adopted in China since 2018, where an MIoT model was used for lung cancer screening campaign. The innovative screening system compares, via a network of suitable processors, the tomographic image of the subcentimetre nodules identified in a patient with those of an image archiving system. In this way, an effective screening model was created based on radiological evaluation in real time and on simultaneous comparison with previous exams. By developing PNapp5A, an application based on the five-step assessment of pulmonary nodules, it was possible to improve the early diagnosis of pulmonary nodules using big data-based management technologies.^12–14

This artificial intelligence system has been adopted in China in 900 centres adhering to the Chinese Alliance Against Lung Cancer (CAALC) and, according to data from Fudan University - Zhongshan Hospital, has proved to be effective in the early diagnosis of pulmonary nodules, lowering the average age of diagnosis from 63 to 50 years, and obtaining that of all patients undergoing surgery, 60.3% had undergone early diagnosis with this model. Based on this experience, Dr Chunxue Bei introduced the term ‘human–computer multidisciplinary team’, emphasizing the interaction and collaboration between the physician and the artificial intelligence, the latter considered almost an external entity with its identity and autonomy. This new approach facilitated the standardization of screening, diagnosis, and treatment of early-stage lung cancer and complex cases of nodules of an indeterminate nature.^2,12,13

From these first experiences, we anticipate that the applications of the metaverse in medicine can be multiple with an apparently unlimited potential.

In the field of rehabilitation and psychology and psychotherapy, perspective for applying technologies related to the metaverse seems very real. Techno Village Rehaveware is a VR rehabilitation service focused on recovering impaired motor function in patients with brain diseases such as stroke, Parkinson’s disease, and brain surgery. Through the interaction with objects belonging to the IoT typology (smart balls or globes), it is expected that the patient’s motivation to exercise would increase, as well as the clinical results. In the future, thanks to the possibility to create virtual entities parallel to the real world, its use is expected to be extended to the psychological treatment of patients with dementia, as well as children and adolescents suffering from family violence or other severe mental disorders.^2,3

Another field of application of artificial intelligence extended to the metaverse is surgery. In Lisbon, at the Breast Unit of the Champalimaud Foundation, the Portuguese surgeon Dr Pedro Gouveia and his Spanish colleague Dr Rogelio Andrés-Luna, thanks to the metaverse, performed an operation simulating of being in the same operating room, despite of being 900 km away. Dr Gouveia wore special AR glasses, namely Hololens.² Not only could he see the patient in front of him, but he also had patient’s diagnostic images and clinical information projected onto the appropriate lenses. In this context, 5G technology proved to be essential, capable of overcoming the limits (such as latency time) of 4G technology.

Artificial intelligence and VR appear to be very promising in the field of surgery. Not only would they allow interventions to be performed remotely, but they would allow operators to be completely immersed in the intervention itself. In this sense, AR would play a fundamental role. The surgeon will wear smart glasses able to inform him in real time of any vital parameter changes and to provide him with all the information necessary to improve his performance, without the need to take his eyes off the patient and the operating field.

Moreover, the possibility of accessing the VR would also represent a valid tool in surgical planning. Thus, it offers the possibility of performing not only 3D reconstructions of the target organ but, providing a 4D vision, also the possibility of navigating inside of it and analyse any possible anomaly. It would also be possible to simulate the operation in VR on an avatar identical to the patient to be operated on, acquiring experience before performing the real operation in the operating room.^1,2–4,14

Perspectives are also encouraging in the field of clinical research. Thanks to the IoT, through intelligent devices, we might be able to reproduce virtual patients, with characteristics similar to real ones, to treat with innovative therapeutic protocols, considerably speeding up achievement of results of clinical trials and thus reducing time and costs of trials.

The metaverse in medical education and training

The role of the metaverse in healthcare education should also be emphasized. Through VR and through the use of intelligent objects, a series of very useful possibilities in teaching are available. There is the possibility to perform anatomical studies on virtual models of real organs, being able to dissect them, to feel the consistency of the tissues, to evaluate the anatomical relationships with the surrounding structures, up to enter into their interior to better evaluate their anatomical characteristics. It is possible to perform surgical procedures in conditions equal to real ones or to simulate clinical or diagnostic activities on virtual models. In the same way, simulation models of physiological or pathological mechanisms can be recreated. For example, the BRM all-in-one machine adopting holographic emulation technology was used to show students the mechanism of cigarette smoke inducing lung cancer. This pioneering pedagogical practice produced sensational effects as students observed closely the alveolar damage caused by smoking and its relationship to the onset of lung cancer. By means of virtual patients with characteristics similar to real patients, it is possible to safely carry out treatment paths, from the anamnesis, to the diagnostic procedure, to the patient’s therapy, representing a valuable teaching tool for the student to approach the clinical situation in total security and still gaining valuable experience.

The possibility of sharing studies with colleagues located in different parts of the world favours group activity and teamwork in an interactive way, from the reconstruction of anatomical systems to the collegial discussion of clinical cases. Sharing experiments gave very positive results. Furthermore, students can be trained to quickly learn various therapeutic techniques as if they were present in clinical practice, such as magnetic navigation, or even procedures such as endoscopy or intubation. Naturally, as already highlighted, the training of young surgeons is facilitated with the possibility to perform simulated operations on virtual patients and to guide the movements of a student even remotely.^1,14

The metaverse would also favour direct experiences regarding the state of patients. In 2006, a virtual psychiatric unit was created, with the aim to simulate the virtual experience of visual and auditory hallucinations, to give to trainee doctors the opportunity to learn more about these pathological phenomena by experiencing them directly.¹⁵

Conclusions

The connection between artificial intelligence (in its varied forms) and the metaverse is very complex, and often various models of artificial intelligence can be the basis and the essential components of an innovative model of the metaverse. Transition from the various forms of artificial intelligence to the metaverse is not immediate. The potential of the metaverse appears unlimited, but to reach its effective utility and reliability in the field of medicine, future developments are expected.

Funding

None declared.

Data availability

No new data were generated or analysed in support of this research.