Collaborative 360° virtual reality training of medical students in clinical examinations

ABSTRACT

Simulation-based training in computer-generated environments has always played an important role in clinical medical education. Recently, there has been a growing interest in using 360° videos of real-life situations for training in health professions. Several studies report positive results from using 360° Virtual Reality for individuals, yet there are currently no studies on collaborative 360° Virtual Reality training. In this paper, we evaluate how 360° Virtual Reality can support collaborative training in clinical medical education. The study population consisted of 14 medical students in semester 5 of their Bachelor’s programme. The students were divided into three groups before watching and annotating a 360° video of an authentic learning situation inside a collaborative immersive virtual reality space. The original video shows a problem-based examination of the collateral and cruciate ligaments of the knee performed by students under the supervision of a professor. After training in collaborative 360° Virtual Reality, students then had to perform the same tests in a physical examination. The students’ performance was subsequently evaluated by a professor with expertise in knee examinations. The results show that 12 out of 14 students received a score of 2 for one or more tests, thereby meeting the required learning objective. One student received a score of 1 and one student did not perform any of the tests. The students actively use the tools provided by the software and different communicative strategies when working collaboratively in 360° Virtual Reality, which enables them to perform the tests in the physical examination by transferring their constructed knowledge. The results indicate that our pedagogical design in collaborative immersive 360° Virtual Reality can become a relevant addition to face-to-face clinical medical training.

Introduction

During the COVID-19 pandemic, medical education programmes experienced difficulties in supporting and giving access to the ‘immersive nature of medical education’ [1], p. 1995). As medical education is heavily reliant on direct human interaction and hands-on clinical training, the sudden and widespread requirement for social distancing measures had a profound impact on medical education. In clinical medical education, computer simulation-based training has always played an important role [2–10] as students can practise their skills in a risk-free environment and without harming human subjects. The need for simulations only increased during Covid-19, and the possibility of a new pandemic is paving the way for digital health education. Currently, simulation-based training uses computer-generated environments, which do not convey the situated and immersive nature of medical education in all its multimodal detail [11]. However, there has been a growing interest in using 360° videos of real-life scenarios for learning activities [12,13]. With 360° video, teachers and students in health profession education are no longer confined to viewing a video on a flat screen, but instead, the 360° videos can be projected in a Head-Mounted Display (HMD) [12]. The use of 360° videos also offers new opportunities for working with authentic learning situations, which according to [14] prepares the students for dealing with new real-life situations and provides access to the practice they will be a part of at a later point. Rachul et al. [11] also suggest that health profession education should focus on all the different modes of interaction, as new technologies shape the context of medical work. Another positive outcome of using 360° Virtual Reality is a higher degree of involvement from the students while watching the video [14]. Several studies report positive results from using 360° Virtual Reality for individuals within health profession education [14–17]. While some studies use individual 360° Virtual Reality as the foundation for later group discussions [18], there are currently no studies on collaborative training within 360° Virtual Reality. The study presented in this paper advances our knowledge about how 360° Virtual Reality can support training in a collaborative and Problem-Based Learning (PBL) space in clinical medical education. We are thus using a new technological medium to develop case PBL [19], allowing for more sophisticated interaction with authentic cases from clinical medicine. We report on a pilot study in which we investigate the potential of using a collaborative 360° Virtual Reality learning space for clinical medical training and examinations. In our previous work we have unfolded the potential of the medium in a PBL setting by looking at the students interaction in collaborative 360° Virtual Reality [20]. In this paper we are interested in understanding whether it is possible to transfer the knowledge gained from working collaboratively in immersive 360° Virtual Reality to a physical setting. We view transfer as situated, meaning that the knowledge constructed in 360° Virtual Reality is not necessarily present in the following examinations, but must be actively contextualised by the students through their participation and interaction [21]. Our results indicate that collaborative 360° Virtual Reality has great potential for supporting medical students’ knowledge and skills retention. While this pilot study does have limitations, we can use it as an extreme case [22] to explore different pedagogical designs and indicate the future direction of collaborative 360° Virtual Reality in clinical medical education.

Methods

Study design

To study how immersive 360° Virtual Reality can be used as a collaborative learning space for training medical students in clinical examinations, we designed a problem-based learning case [19] in which groups of medical students had to collaborate in a collaborative immersive learning space. In this digital learning space, groups of students interacted in/with a 360° video of an authentic learning situation mediated through collaborative 360° Virtual Reality. The students were able to use different tools provided by the software to work collaboratively with/in the 360° video (seedescription of the software below). After the collaborative training in collaborative 360° Virtual Reality, the students performed a similar examination as the one performed in the original 360° video in groups of two or three. This allowed us to study the effectiveness of the training in terms of the students’ knowledge and skills retention. A visualisation of the design of the problem-based learning case is shown in Figure 1.

Figure 1.

A visualisation of the problem-based learning session.

The collaborative 360° Virtual Reality training activity was based on a 17-minute-long unscripted 360° video showing a professor and two students (see Figure 2) in a room at a hospital, performing a knee exam. This video was part of a longer session lasting 100 minutes, but we decided to limit the collaborative Virtual Reality training session to a shorter and more focused section of the original video. Compared to traditional case PBL, this session was more open-ended and the students had to negotiate how to work together in the collaborative virtual space, and were given primary responsibility for defining their learning strategies and goals.

Figure 2.

(a) picture from the 360° training video. (b) The technical setup outside of the CAVA360VR space. (c) The technical setup inside the CAVA360VR space.

The original 360° video features novice students performing an examination of the collateral and cruciate ligaments of the knee for the first time. The students in the original video have no experience in performing this type of examination. Therefore, they are also identifying and solving a problem together with no prior professional experience. In the video, one student examines another student’s knee. Due to their lack of experience with this type of examination, the video shows some failures and uncertainties about how to perform the examination. In the original video, the professor instructs and guides the students by questioning their examination, e.g., how they twist and bend the knee to perform certain tests. We consider this original video to be an authentic setting as the students and professor are together creating a situated learning space, with the situation not being planned or scripted in any way prior to filming. The idea of using authentic (i.e., unscripted) material differs from the very structured rote-learning material that is normally used in clinical education.

To support collaborative learning in 360° Virtual Reality, we used the software prototype CAVA360VR, which was developed by a group at Aalborg University [23,24]. This software enables the students to play and annotate a 360° video while being situated in different physical locations through a networked connection. The prototype developed in Unity using the PUN network utility currently supports up to 20 users in the virtual space at the same time. When immersed in the virtual space, the students are represented by avatars. The avatar consists of a floating head and hands, and on top of each avatar head, the name of the student/user is visible (see Figure 2). The students can use a laser pointer to indicate what they are looking at and referring to, and they can also use the laser pointer as a drawing tool if they need to illustrate something in the video. The software also enables the students to stop and pause the video [24]. The voice of the participants are captured by the microphone in the HMD and broadcast to the other participants. The sound is spatialised so the participants will get a sense of where the other participant is located.

Setting and research participants

In late November and early December 2021, we conducted our pilot study consisting of three experiments in which a total of 14 students (2 male and 12 female) enrolled in the 5. semester of their medical Bachelor’s programme at Aalborg University participated. The students were recruited through an open call for participants in the Learning Management System at the university. As such, we did not have any possibility to balance the composition of gender, or other demographic criteria. The only employed sampling criteria was that the participants did not have any prior experience with collaborative 360° Virtual Reality.

Three students participated in the first experiment using an unedited version of the training session in collaborative 360° Virtual Reality. The students had not received any teaching or been shown how to perform clinical examinations before the experiment, nor had they had any substantial experience with collaborative 360° Virtual Reality. For the second experiment, six students participated also using an unedited version of the training setup. This group of students had all previously participated in the clinical examination course that forms part of the curriculum for semester 5 of the medical Bachelor’s programme. Due to COVID-restrictions, the teaching consisted of the students watching a professor perform the different clinical tests over a webcam feed. None of the students had therefore tried to perform the tests themselves. Five students participated in the third experiment, this time with a edited version of the original video, in which a number of questions were embedded. This meant that one of the students received four audio-visual prompts throughout the 17 minute long video, asking them to pause the video and reflect on the question, before resuming the video. Here, it is important to note that the playback state is shared across participants in the CAVA360VR prototype, meaning that if one student pauses, the video pauses for all students in VR. This enabled us to understand how a collaborative 360° Virtual Reality space can be designed by trying out different setups. Like group two, this group had also received prior teaching in clinical examinations but had not tried to perform the tests.

Prior to the training, all of the students were sent to different physical rooms and given an HMD and a short presentation of the different buttons on the controllers, so they had a basic understanding of the hardware and software (see Figure 2).

There was no time limit for the training because we were interested in exploring how much time the students spent preparing for the examination. The time spent by the three groups inside the collaborative 360° Virtual Reality space viewing and interacting with the 17-minute clip was as follows: 40 min 17 sec for Group 1, 36 min 28 sec for Group 2 and 49 min 36 sec for Group 3.

After the training ended, the students were allocated to an examination room, where they performed a physical examination of the collateral and cruciate ligaments of the knee (see Results).

Data collection

As collaborative 360° Virtual Reality is a new digital learning space for clinical medical education, we decided to collect multimodal video data [11]. All of the students’ actions and interactions both inside the collaborative 360° Virtual Reality and in the physical room were recorded using video cameras and screen capture tools. The reason to collect data this way was to ascertain whether the students had any gestures or body movements in the physical space that were not expressed in the virtual space.

To record the students’ actions during the physical examination, we used one camcorder placed at the foot of the bed, two 360° cameras placed at each end of the bed, and a GoPro mounted on the ceiling. We used several cameras to make sure we had the best possible overview of how the students performed the clinical tests (see Figure 3).

Figure 3.

An overview of the different camera angles used to capture the physical examination.

To record audio, two students had a mobile microphone attached to their shirt and a 360° spatial recorder was placed next to the bed.

Data analysis

In this study, we focused on how the students accomplished the different clinical tests after collaboratively training in collaborative 360° Virtual Reality to determine whether their performance met the learning objectives required by the University. We focused on tests of the medial and lateral collateral ligament on a stretched knee and tests of the anterior cruciate ligament on a bent knee at 90°. We did not expect the students to know how to perform a test of the posterior cruciate ligament because this was not mentioned in the training video. The students’ performance was assessed by a clinical professor with expertise in knee examination and a grade system was developed for this purpose (see Table 1). The clinical professor and the first and second author of this paper reviewed the entire video corpus from the physical examinations to assess the performance of the individual student. In addition, as part of reviewing the video data we also identified how the students used tools provided in CAVA360VR and further, we traced the communicative strategies that the students used to plan how they would perform the physical examination (see Table 2). The results (Table 2) are based on the careful transcription of the video data using DOTE [25] and coding of the different use of the software tools and communicative strategies afterwards.

Table 1.

The time spent on training in immersive 360°VR by the three groups, and the time spent on the physical examination in groups of two or three. The students’ performance in the clinical tests was graded (2 = the performance met the objective learning requirements, 1 = the student performed the test, did not meet the objective learning requirements, 0 = the student performed the test incorrectly, - = the student did not perform the test).

Group 1 (3 students)	Time spent in immersive 360°VR	Time spent on the physical examination	Medial collateral ligament	Lateral collateral ligament	Anterior cruciate ligament
Student A	40 min. 17 sec.	5 min. 45 sec.	2	2	-
Student B			–	–	–
Student C			2	2	2
Group 2(6 students)
Student D	36 min. 28 sec.	4 min. 54 sec.	–	–	2
Student E			2	2	–
Student F		4 min. 30 sec.	1	1	2
Student G			2	2	2
Student H		3 min. 42 sec.	–	–	1
Student I			2	2	2
Group 3 (5 students)
Student J	49 min. 28 sec.	16 min. 21 sec.	–	–	2
Student K			1	1	2
Student L			2	2	–
Student M		7 min. 17 sec.	1	1	2
Student N			1	1	2

Table 2.

The use of software tools and communicative strategies when performing the physical test, by group.

	Group 1	Group 2	Group 3
The software tools
Visualize with laser	Yes	Yes	Yes
Drawing	Yes	Yes	Yes
Manipulation of the timeline (rewind/fast-forward)	Yes	Yes	Yes
Pause/play	Yes	Yes	Yes
Write	Yes	No	No
Communicative strategies
Answer the professors’ questions	No	Yes	Yes
Summarizing	Yes	Yes	Yes
Discussing/debating	Yes	Yes	Yes
Preparing and planning how to perform the physical examination	Yes	Yes	No

Results

Results from the physical examination

The results from Table 1 show that in the test of the medial collateral ligament, six students received a score of 2, four students received a score of 1 and four students did not perform the test (-). In the test of the lateral collateral ligament, six students received a score of 2, four students received a score of 1, and four students did not perform the test (-). In the test of the anterior cruciate ligament, nine students received a score of 2, one student received a score of 1, and four students did not perform the test (-). No students performed the test incorrectly. In total, 12 out of 14 students met the required learning objectives in one or more tests. A possible reason as to why some of the students did not perform the examinations can be found in our instructions and the design of the physical examination. The students performed the examinations in dyads and triads, and we did not explicitly state that each of them should perform the examinations.

The students spent varying amounts of time on the clinical examination, with the shortest time at 3 min 42 sec and the longest time at 16 min 21 sec. In Group 1, 2 out 3 students met the learning requirements in relation to these clinical tests. In Group 2, 3 out 4 students met the learning requirement. In Group 3, 3 out of 5 students met the learning requirement. These results indicate that there is no significant difference between the groups, even though they were immersed in 360° Virtual Reality learning space in different periods of time. Further, there is no indication that pre-training or exposure to clinical examinations prior to the pilot study is influencing the results.

Results from the training in collaborative 360° virtual reality

The results from the training in the collaborative 360° Virtual Reality learning space are presented in Table 2. The results show that the students actively used different tools in the software CAVA360VR and communicative strategies to construct a shared understanding of the information presented to them in the collaborative 360° Virtual Reality learning space. These tools and communicative strategies enabled them to perform the physical examination after the collaborative activity in 360° Virtual Reality.

The results show that Group 3 did not prepare or plan by the end of the session, but instead extended their learning session to the physical examination. This illustrates that the students have different communicative strategies and different understandings of how to perform the physical examination. Group 3 spent the most time in collaborative 360° Virtual Reality, which could be a consequence of the questions prompted in the digital learning space throughout the session. The questions forced the students to stop and discuss, whereas Group 1 and 2 had to negotiate when to stop the video based in their own experience. Interestingly, Group 3 spend the most time in collaborative 360° Virtual Reality learning space, but 3 out of 5 students did not meet the learning requirement for the examination of the medial collateral and lateral collateral ligament. This could be the result of Group 3 extending their learning session to the physical space, with the other two groups viewing the examination as an evaluation of the learning session. When conducting the pilot study, groups were broadly asked to perform a knee exam similar to the one in the video. It was therefore not clearly stated whether the physical exam was an extension of the learning session or an evaluation of their collaborative training.

Discussion

In this pilot study, we evaluated the potential for transferring knowledge and skills from a collaborative 360° virtual reality learning space to a physical clinical examination. In addition, we have assessed how the students used the collaborative 360° Virtual Reality software CAVA360VR and the students’ communicative strategies in this digital learning space. To do this, we recruited students in the 5^th semester of the medical Bachelor’s programme. This group of students met the inclusion criteria, i.e., students with no experience in clinical practice and with a basic knowledge of anatomy [26]. The design of the Problem-Based Learning case allowed the students to learn clinical skills that are normally taught through physical training in clinical examinations. The results show that most of the students were able to perform the three tests even though they had never done it before and had only seen it performed by an instructor. Even students who had not received any prior teaching were able to meet the objective learning requirements. Despite some of the students not performing one or more of the tests, they still participated in the examination by guiding and helping the other student(s), for example Student B in Group 1. This correlates with other results, e.g., that by collaborating students can learn skills and competencies related to communication, collaboration and problem-solving in order to construct a shared understanding of the information presented to them [27]. By using and incorporating the software tools in CAVA360VR in their learning strategies, the students used the collaborative 360° virtual reality learning space to analyse the original video in a collaborative way. Social and communicative skills are advantageous in this digital learning space because the training video does not always allow the students full access to see the different ways of performing the clinical examinations, e.g., sometimes the students cannot see what is going on due to the angle of the camera or because it is hard to hear what the students in the original video are saying. By working collaboratively, they can integrate each other’s knowledge and experiences to fill in the gaps and construct a better understanding of the physical examination performed in the original video. The students used their prior knowledge of e.g., anatomy to help and guide each other both during the training in the collaborative virtual space and afterwards in the physical examination. This is a positive outcome that demonstrates that the students can use their knowledge in an active way in the collaborative 360° virtual reality learning space.

These results (Table 1) give us confidence that our training design with the use of collaborative immersive 360° Virtual Reality has the potential to become a relevant substitute for, or addition to, face-to-face clinical teaching and training of medical students. Collaborative 360° Virtual Reality offers the students a more sophisticated and advanced learning space, where they can immerse themselves in the video material alongside their fellow students. In our follow-up interviews with the students, many of them expressed an interest in having a library of 360° Virtual Reality cases. This would allow them to return to the different cases and practise their skills and competences with fellow students. The relatively small number of students in the pilot study is of course a limitation, but the results indicate that collaborative 360° virtual reality can provide a learning space in clinical medical training. In the future, we plan to expand the pilot study and increase the number of students and bring together a more balanced mix of genders.

Integrating collaborative 360° Virtual Reality into existing curricular and training programmes in medicine is of course a major task; it will require investments in technology, but most importantly it requires sound pedagogical framings of how to design for collaboration in 360° Virtual Reality, something which is currently missing [28]. The potential of integrating collaborative 360° Virtual Reality in medicine is that students can get access to authentic situations from clinical practice, which can support their learning. Also, if a new pandemic happens then medical institutions can still provide access to the immersive nature of medical training. Another benefit of using collaborative 360° Virtual Reality is that students can access the situations again and again, without constraints of time and space – and that they are able to train their communication skills with fellow peers, or potentially with other disciplines within the healthcare professions.

As noted in the introduction, the use of computer-generated 3D simulations is the primary type of simulation-based training currently used in medical education [29,30]. This type of simulation can be used for training, where students can practise new skills in a simulated space that allows for corrections, repetition and risk-free failure, and offers the chance to interact with environments that would otherwise be out of reach due to economic or geographic concerns [31]. Despite all of the positive outcomes this technology has contributed to the field of medical education, there is still a need for more platforms supporting collaborative learning while still supporting the immersive nature of medicine [1]. This is especially relevant if a new pandemic would occur. With our training design and the authentic problem-based learning case, we offer a new digital learning space that enables medical students to work collaboratively while situated in different locations at home or in the hospital. It would even be possible to collaborate with students from other countries using CAVA360VR. The research is still at an early stage, but we see a potential and an opportunity to fill a gap in medical training when face-to-face training is not an option. As mentioned by the students, there is still potential to improve the usability of the software and the placement of the cameras recording the original data. The results have given us an understanding of the strength and weaknesses of collaborative 360° Virtual Reality, and we are now ready to produce new training material based on the knowledge gained from the three experiments. We plan to test the training setup on a larger scale, which will provide us with more data and help us develop the digital learning space further. In addition, we want to develop the user interface of CAVA360VR (e.g., use different colours for the avatars, a toggle for the laser pointer, etc) to determine whether that makes a difference for the students. Most importantly, we want to integrate the possibility to jump between different camera positions in collaborative 360° Virtual Reality to better support the students in seeing and experiencing what is going on in the original video. A core research question we are now pursuing is how to design for collaborative problem-solving in 360° Virtual Reality, e.g., how to support turn-taking and how to use questions for prompting dialogue and reflection, etc. This can support future integration of this type of digital learning space in medical training internationally. In addition, it is important that more conditions are tested, e.g., how the collaborative activity can change if one student is given extra information or asked questions that the student would have to rely on the other students. Basically, we pursue a thorough understanding of how to design for collaborative learning in 360° Virtual Reality and what students can learn through participation in this digital learning space.

Funding Statement

The author(s) reported there is no funding associated with the work featured in this article.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Authors’ contributions

J.D. and D.V.L. wrote the main manuscript text and S.R. and L.P. made substantial contributions to the text. D.V.L. prepared the figures and tables. L.P. collected and analysed the data. All authors reviewed the manuscripts.

Ethical approval

All data used in this project were collected according to the rules and GDPR regulations provided by Aalborg University, and the research was approved by the Contract Unit at the university (2021-068 -01,827). Informed consent was obtained from all students who participated in the three pilot experiments. The data were used for educational purposes, and approval from the local ethics committee of the North Denmark Region was therefore not required.