Use of Augmented Reality to Assist Teaching for Future Perfusionists in Extracorporeal Technology

Abstract:

The aim of this study was to foster the better perfusion education when providing extracorporeal circulation (ECC) technology for future perfusionists. For this purpose, we have developed an augmented reality (AR) program for ECC students. Currently, the cost of equipment and its simulator is high. Furthermore, it is desirable for ECC students to practice at any time. AR describes user experiences that add 2D (plane detection) or 3D elements to the live view from a device’s camera in a way that makes those elements appear to inhabit the real world. We can use these technologies to create AR experiences using the back camera of a smartphone or tablet. We can also build our own instrument with custom visualization and data analysis. Although AR technology may not be new, its potential in ECC student education is just beginning to be explored. Unlike other computing technologies, AR interfaces offer seamless interaction between the real and virtual worlds, a tangible interface metaphor, and a means for transitioning between real and virtual worlds. Here, we have shown our experiences of cost-effective AR technology for future perfusionists.

DESCRIPTION

In the past two decades, there has been an increasing use of computer-aided training to help future perfusionists improve technical skills and increase their progression rate during the training process (1). The role of simulators of extracorporeal circulation (ECC) as complementary devices for traditional learning methods is now a fundamental part of today’s training and learning activities. Patient hemodynamic models can be created from the fluid mechanics element, irrespective of patient’s age. However, replicating patient variation could become very complex in such situation. Although 128 parameters were needed to be completed in the past, Ninomiya et al (2). developed a new system, resulting in only 17 parameters needed. Specialized instructor’s knowledge is required in such cases as the determination procedure of these parameters becomes more complicated. Similar systems can be applied in medical settings to assist medical trainees to acquire skills in a safe environment with less potential risk to patients (3). Simulation technology is increasingly used for training medical professionals and is anticipated to become more relevant in the setting of restricted clinical training hours and heightened focus on patient safety.

Augmented reality (AR) environments add convenience and flexibility and increase the ability to scale and distribute simulations widely with lower costs. As with traditional simulation, the degree of fidelity is driven by the learning objectives (4). AR is more likely to be successful if it is systematically integrated into a well-thought-out education and training program which objectively assesses technical skills improvement proximate to the learning experience. In other fields, including the AR neuronavigation system (5) and AR in surgery (6), AR technology is increasingly used. However, this report may be the first one in AR for future perfusionists because there were no similar reports dealing with it as follows: previous reports were retrieved from the scientific database Web of Science Core Collection and PubMed, and the search terms used were “cardiopulmonary bypass, ECC, heart-lung machine, perfusionist, and AR” regarding articles published during the whole timespan covered. AR is increasingly used across a range of subject areas in health care education as health care settings partner to bridge the gap between knowledge and practice (7). AR presents a semi-true and false image, which is the combination of real and virtual; some people commonly describe it as the “third eye” (8). Now, we have created AR for future perfusionists in extracorporeal technology as follows: we have used the MERA Heart Lung Machine HASⅡSystem (Senko Medical Instrument Mfg. Co. Ltd., Tokyo, Japan) as a real world. To develop AR application (app), we have used Apple’s software development kit (Apple Inc., Cupertino, CA), including ARKit2.0, Xcode10, and Swift4.2 (Table 1) (9). Although we are not IT professionals, we could develop them. The software has been run on iPad with iOS12. Using the software, we have developed a new system for teaching how to use AR when manipulating an artificial heart-lung machine.

Table 1.

Comparison AR technology with ECC simulator.

	AR Technology	ECC Simulator
Definition	Combination of real and virtual world on a mobile device	Combination of real and virtual world on a fixed ECCSIM® machine (Senko Medical Instrument Mfg. Co. Ltd., Tokyo, Japan)
Software	Apple® Xcode10, ARkit2.0, and Swift4.2 (free software)	ECCSIM®software included in the system
Hardware	Apple® Macbook Pro (US$1,500) and iPad (US$600)	ECCSIM® (US$40,000)
Availability	Pneumogram	Hemodynamic monitoring with pump control

Definition, software, hardware, and current availability are compared, especially in terms of cost.

To enrich the physical world with augmentations, a software app that uses one or more of the different hardware components must be installed on the device. There are two primary AR software implementation types: marker-based and markerless AR. Marker-based AR uses 2D or 3D images such as a QR code or a physical object, which can be recognized by a software app (10). When the AR software app receives input from the marker or object, it generates the augmented virtual content and projects this information onto the recognized object. In this study, we have used marker-based AR. We have integrated iOS device camera and motion features to produce AR reality experiences. AR describes user experiences that add 2D (plane detection) or 3D elements to the live view from a device’s camera in a way that makes those elements appear to inhabit the real world. The app is very simple to use. After launching it, the tablet/smartphone camera is activated with a yellow marker in the center of the screen, and AR software tracks a correspondence between the real world and virtual space. After the completion of this process, the app is ready to watch pneumogram (respiration waves) on iPad while manipulating a heart-lung machine. To enrich the physical world with augmentations, the app that uses one or more of the different hardware components must be installed on the device. When the AR software application receives input from the marker or object, it generates the augmented virtual content and projects this information onto the recognized object. ECC students perceive that added information really exists within the surroundings; a student is immersed into an enhanced reality, as shown in Figure 1. When operating ECC, students manipulate roller pumps and monitor vital signs. In some instance, unusual events may take place, including a breakdown of the machine (Figure 2). In these situations, AR training may be useful for future perfusionists. As shown in Figure 3, virtual content (pneumogram) is added to the real world (roller pump and controller of heart-lung machine). A hardware device (smartphone and/or tablet) including software is used to make the content visible for a future perfusionist. In this way, real world has been augmented with virtual content. Thus, students can observe both real-world (roller pump and controller) and other information (respiration waves) simultaneously.

Figure 1.

AR world in real world.

Figure 2.

Mobile AR education design framework and AR technology.

Figure 3.

AR in ECC and general anesthesia. (A) AR-generated pneumogram appears on iPad while future perfusionists are manipulating the real machine. (B) AR in ECC in higher magnification. (C) An anesthesia machine without AR image on iPad is seen. (D) An anesthesia machine with AR image (pneumogram) on iPad is seen before initiating artificial heart and lung machine.

DISCUSSION

Educational technology has the potential to offer a safe, suitable, and cost-effective training setting in which whole real-world training tasks can be practiced. In such controlled environments, students can make errors without adverse consequences, and instructors can focus on students. Those learning environments also give students opportunities for just-in-time and just-in-place learning (10). AR is referred to as a promising technology for education and clinical medicine. In this article, we have shown our experiences of cost-effective AR technology for future perfusionists.

In our department, an introduction to extracorporeal techniques through discussion of blood propelling devices, heat transfer, gas transfer, bio-materials, and perfusion physiology. An introduction to basic extracorporeal systems includes simulated cardiopulmonary bypass to learn the assembly, use, and control of the bypass system. Emphasis will be on catastrophe management. We are now on the verge of adopting AR technologies to enhance the education program for future perfusionists. ECC students and instructors can interact with the AR interface in two different ways. Students can freely manipulate the marker in three-dimensional space to receive a different perception. They can also zoom, pan, and rotate the rendered images. On the other hand, the instructors can interact with the student without interruption. One clear advantage is that the instructors can speak to the student at the same time.

Our study has several strengths, including that we can create an AR program with low cost and this program is easy to use. Currently, making additional AR programs where mixed flow of blood and cardioplegia can be visible is in progress. Our study has certain limitations. For example, what those components are and how these components positively affect ECC student motivation and validation remain undetermined (11). Research on AR in education has demonstrated that AR applications designed with diverse components boost student motivation in educational settings. However, most of the research does not define exactly what those components are and how these components positively affect student motivation. Another limitation is that our app runs on Apple’s iOS only, whereas cross-platform development is a popular buzzword. With the consideration to develop natively for a single platform, iOS would be an ideal choice if it is to be used as an internal institutional app. Although Android is equally capable and more versatile in many ways, from a developmental and industrial perspective, Android also has a vastly more diverse blend of devices, OS versions, and varying limitations that can impede development time and testing (12).

In conclusion, our results indicate that future perfusionists may study AR in classrooms because there is an intimate relationship between virtual and physical objects. Furthermore, this AR technology for ECC is cost-effective and relatively easy to construct.

Virtual content is added to the real world. A hardware device including software is used to make the content visible for a future perfusionist. Marker-based AR uses 2D or 3D images such as a physical object (for instance, a wall), which can be recognized by the software application is seen in the real world.

On ECC, future perfusionists manipulate roller pumps and controllers and monitor AR-generated pneumogram as one of the vital signs. In some instance, unusual events may take place, including a breakdown of the machine. In these situations, AR training may be useful for future perfusionists. An established AR pneumogram in this study and examples of the future AR are seen.