Smartphone-based augmented reality patient education in radiation oncology

Highlights

• •Built a smartphone-based augmented reality app for patient education.
• •Shows cancer patients 3D animation of their own radiotherapy plan.
• •Viewable on any iOS/Android tablet/smartphone without specialized hardware/software.
• •Smartphone home viewing improved learning and access.
• •Patient trial showed improved understanding for 95%, decreased anxiety for 60%.

Abstract

We built an augmented reality (AR) patient education application for portable iOS and Android devices that allows patients to view a virtual simulation of themselves receiving radiation treatment. We created software that reads data from the clinical treatment planning system and renders the patient’s actual radiotherapy plan in AR on a tablet or smartphone. The patient's CT simulation data are converted into a 3D translucent virtual human shown being treated with visible radiation beams from a virtual linear accelerator. We conducted a patient study to determine if showing patients this AR simulation improves patient understanding of radiotherapy and/or reduces anxiety about treatment. A total of 75 patients completed this study. The most common plans were 3D breast tangents and intensity modulated radiotherapy lung plans. Patients were administered questionnaires both before and after their AR viewing experience. After their AR viewing, 95% of patients indicated that they had a better understanding of how radiotherapy will be used to treat their cancer. Of the 35 patients who expressed anxiety about radiotherapy beforehand, 21 (60%) indicated that they had decreased anxiety after the AR session. In our single-arm prospective patient study, we found that this simplified low-cost tablet-based personalized AR simulation can be a helpful educational tool for cancer patients undergoing radiotherapy.

Introduction

Cancer patients are often anxious about the prospect of undergoing radiotherapy [1], [2]. They may have misconceptions about radiation and often do not have easily accessible information about radiotherapy treatment [3], [4]. Illustrating the process of radiation treatment is often difficult because of the spatial complexity of radiotherapy delivery. A better understanding of the radiotherapy process could make treatment more tolerable and reduce patient anxiety [5].

Virtual reality (VR) and augmented reality (AR) technology can be more effective at conveying information that requires a three-dimensional understanding of an environment. This novel VR/AR technology is becoming more widely used throughout medicine [6], including for medical research [7], surgical planning [8], medical training [9], [10], and patient therapy [11]. VR/AR is an especially powerful/useful tool for patient education [12], as it has been shown that these immersive virtual experiences can promote improved recall [13] and decrease anxiety [14], [15].

VR/AR technology is well suited for multiple applications in the field of radiation oncology, since the process of designing, planning, and delivering radiotherapy requires a detailed knowledge of 3D spatial relationships between radiation beams, tumor location, and adjacent organs. VR and AR have great potential for use by radiation oncologists, dosimetrists, and physicists; furthermore, VR/AR technology is now becoming more widely used in this field for training [16], [17], treatment planning [18], improving patient compliance [19], and patient education [20], [21]. Some have used a system with 3D glasses and a video display (VERT) [22], [23], [24], while others have used pre-recorded videos filmed with a special 360 degree video camera [25], [26].

We previously performed a pilot study demonstrating the results of a novel VR headset-based system showing patients an immersive 3D simulated animation of their actual radiotherapy treatment plan [27]. In this pilot study, we used DICOM-RT data from the patient's actual radiotherapy treatment plan and created a VR animation of the linear accelerator delivering visible yellow radiation beams aimed at a specific visible target in the patient's semi-translucent body (constructed from their CT simulation) lying on the treatment table. It allows the patient to walk around the room with only a tablet or smartphone and see a “window” into the life-sized 3D virtual radiotherapy treatment room and see the radiation being administered to their virtual body on the table. Although the study was closed early due to the COVID-19 pandemic, we were still able to demonstrate that this educational VR system decreased anxiety and improved patients’ understanding of their treatment plan.

While these headset-based VR systems offer the most realistic, immersive experiences, they require the use of specialized hardware. Additionally, these VR educational sessions are usually performed only in the clinic with staff support. In the post-intervention survey of our original study, 70 % of participants stated that they would be very interested in having a version of this experience that they could take home and watch again or show family and friends.

In this follow-up study, we built an augmented reality (AR) patient education system that can run on Android or iOS mobile platforms. This AR system allows the user to experience a 3D animation without the need for specialized VR hardware. Patients can load this demo onto their own smartphone for repeat viewing later at home, which can reinforce learning. We built this new AR system as an alternative patient education platform that may facilitate more widespread adoption.

In this paper, we report the results of our patient study using our new tablet-based AR patient education system. The purpose of our study was to determine whether this AR tool can improve patients' understanding of how radiotherapy will be used to treat their cancer, or alleviate their anxiety about undergoing radiotherapy.

Materials and methods

Technical development

First, the patient's radiotherapy plan is exported from our clinical treatment planning system in standard DICOM-RT [28] in a de-identified format. The plan files can then be uploaded to our custom NodeJS secured web server via the administration dashboard (Fig. 1) where they are processed by our custom python library and converted into a 3D treatment animation. All DICOM-RT patient structures (e.g., target volumes, normal organs) are converted into 3D geometry meshes. Radiotherapy beam information is extracted from the DICOM-RT file (jaw positions, MLC positions, gantry angle, collimator angle, table angle, monitor units) for all beam control points. The beam sequence data are used to generate an animation of LINAC movements and beam shape models to simulate the entire treatment sequence. The animations are applied to a 3D model of a linear accelerator with a fully movable patient table and gantry head (gantry & collimator angles) and adjustable multi-leaf collimator (MLC) leaves. Each beam in the treatment plan is played in sequence in real-time assuming a nominal 600 monitor units per minute delivery time. Gantry movement is modeled as smoothed arcs for dynamic conformal arcs or volumetric modulated arc-based therapy. MLC leaf motion is modeled for each control point, which dynamically changes the shape of the visible radiation beams in real time (Fig. 2). The animation is saved into two standard 3D file formats so that it can be viewed on any iOS/Android tablet/smartphone without the need for any special app or software download. We used the glTF [29] file format for Android devices, and we used USDZ [30] file format for iOS devices.

Fig. 1.

Browser-based control panel. First, DICOM-RT data from the radiotherapy treatment planning system are uploaded. Next, target and anatomic structures are selected for display, and color/transparency settings can be adjusted. RT beam data are decoded to be played in real-time on a virtual linear accelerator. Finally, all data are converted and packaged into both iOS and Android augmented reality display formats. Left upper lobe gross tumor volume (red), lungs (gray), heart (purple). (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

Fig. 2.

Augmented reality depiction of radiotherapy treatment delivery. Example showing a conformal radiation beam directed at a left upper lobe lung tumor.

Once the animations are generated, each AR plan is listed on the administrator dashboard web portal (Fig. 1) identified by an anonymized six-character ID. Selecting a patient from the dashboard brings up a user interface where the animation can be previewed and the animation appearance can be customized. The color and transparency of each patient structure and the beam can be modified by selecting from a list of presets or using a color picker. The AR animation can be retrieved on any tablet or smartphone browser via a unique URL or by a QR code (containing the embedded URL). Scanning the QR code or typing in the URL on any iOS or Android device will bring the patient to a web view where the 3D animation is rendered using the model-viewer JavaScript library. A button at the bottom of the web page allows the patient to launch the augmented reality view that uses the rear-facing device camera to show the animation as a life-sized cartoon in the patient’s room. On Android devices, the glTF animation is sent to the ARCore [31] scene-viewer API. On iOS devices, the USDZ animation is sent to the ARKit [32] Quick Look API. Each respective API handles the augmented reality tracking and rendering of the treatment animation. The final product is a real-time animation of the patient's radiotherapy treatment plan delivered by a linear accelerator (Fig. 2).

When the user points the tablet or smartphone camera toward the center of the room, they will see an AR view of a life-sized 3D animation of a linear accelerator that appears to be in their room (Fig. 3). Holding their smartphone or tablet, they can walk around the virtual treatment room and see the treatment process from different viewpoints around the room. A full-scale 3D rendering of the relevant part of the patient's body is shown in position on the treatment table with a translucent body contour so that the target volume and selected internal normal organs can be seen. Each beam in the treatment plan is played in sequence in real time and depicted as visible yellow light rays delivered from various angles, dynamically shaped using MLCs to conform to the target volume while avoiding adjacent normal structures.

Fig. 3.

Example smartphone views of our AR system. (a) Web View. On initial webpage load, a 3D preview of the radiation treatment delivery is shown that can be rotated (one finger) zoomed (two finger pinch). (b) Augmented Reality View. By pointing the smartphone rear-facing camera around the room, the user sees an AR animation of a linear accelerator delivering radiation treatment that appears to be located in the patient's living room. (This example AR demo is available at: https://liamjwang.github.io/radoncxr-example/ Can be viewed in the Safari browser (iOS) or Chrome browser (Android or iOS), without any special software download.)

Experimental study design

This study was a single-arm single institution prospective clinical trial, and was approved by our hospital's Institutional Review Board.

Patients were recruited for participation via a research flier from a single radiation oncology clinic. Patients over age 18 planning to receive radiotherapy were eligible candidates for this study. Patients were excluded if they had vision or hearing impairment, or if they had a known history of vertigo, motion sickness or vergence-accommodation conflict associated with 3-D media headsets.

Study patients participated in one research session that lasted approximately 30 min. The research session took place after the patient's radiation plan was finalized but prior to the start of radiotherapy. The patient’s treatment plan was made available on a secured password protected encrypted internet server, each with a unique URL embedded in a QR code that was printed and given to the patient for future reference. A clinical researcher was present during the viewing to provide explanations to the patient during the experience. Ambulatory patients were encouraged to walk around the room to see the linear accelerator setup from different viewpoints, and to move up close to the treatment table to see details inside the translucent body as the beams were being delivered.

Evaluation

We designed a questionnaire to ascertain each patient's current knowledge about their cancer, their understanding of how radiotherapy treatment works, and their anxiety level regarding the prospect of undergoing radiotherapy. We used a standard numerical 5-point Likert Scale, 1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, 5 = strongly agree. Participants completed the same questionnaire both before and after the VR experience, and again at the end of their treatment course to determine if the AR visualization affected patient sentiment over time. Since our results involved surveying the same patients over time, we used repeated measures (paired) t-tests to assess the statistical significance of any differences in each patient's answers to the questions. This test was selected as a more robust statistical test to eliminate any between-patient factors that might otherwise influence results.

Results

From May 2021 through January 2023, 79 patients initially consented to enroll in our study. Of those enrolled, 1 patient subsequently did not undergo radiotherapy and 3 others did not complete all questionnaires. A total of 75 patients completed the study (Demographics in Table 1). The majority of radiotherapy plans were 3D (65 %) and IMRT (25 %), but we also had 6 stereotactic body radiotherapy (SBRT) plans and 1 stereotactic radiosurgery (SRS) plan. The most common disease sites were breast (49 %), lung (12 %), and prostate (8 %). Twenty-three percent of cases were for palliative radiotherapy to miscellaneous metastatic sites.

Table 1.

Patient Demographics.

	n	%
Age (mean)	64.4
Female	50	(67)
Plan Type
3D	49	(65)
IMRT	19	(25)
SBRT	6	(8)
SRS	1	(1)
Disease Site
breast	37	(49)
lung	9	(12)
prostate	6	(8)
rectal	3	(4)
brain	2	(3)
head & neck	1	(1)
other	17	(23)
Total	75	(100)

IMRT = intensity modulated radiotherapy, SBRT = stereotactic body radiotherapy, SRS = stereotactic radiosurgery.

Table 2 shows the results of the pre- and two post- intervention questionnaires. A total of 71 participants (95 %) indicated that they “agree” or “strongly agree” that the AR viewing gave them a better understanding of how radiotherapy will be used to treat their cancer. Of the 35 patients who expressed anxiety about radiotherapy beforehand, 21 (60 %) indicated that they had decreased anxiety after the AR session. After the AR session, 61 participants (81 %) stated they “agree” or “strongly agree” that they had a good understanding of how they would feel when lying on the treatment table, compared to only 44 (59 %) before the session. The number of participants indicating that they understood why radiation might cause side effects increased from 57 (76 %) to 63 (84 %). While most participants felt they had a good understanding of the location of their cancer even before the session (96 %), more participants expressed an understanding of the size of their cancer after the AR session (70 (93 %)), compared to before the session (59 (79 %)).

Table 2.

Patient Questionnaire.

		Before	After	Finish Day	Before-After	Before-Finish
	Question	mean(sd)	mean(sd)	mean(sd)	Differences(p-value)
1	I understand where the cancer is located in my body.	4.61 (0.76)	4.87 (0.38)	4.88 (0.52)	0.26 (0.003*)	0.27 (0.005*)
2	I understand the size of my cancer.	4.07 (0.99)	4.53 (0.72)	4.56 (0.84)	0.46 (<0.001*)	0.49 (<0.001*)
3	I understand how radiation beams will be aimed to treat my cancer.	3.88 (0.96)	4.76 (0.54)	4.84 (0.55)	0.88 (<0.001*)	0.96 (<0.001*)
4	I understand why radiation beams may give me side effects.	4.04 (0.89)	4.24 (0.95)	4.54 (0.83)	0.21 (0.009*)	0.51 (<0.001*)
5	I understand what I will feel like when I am laying on the treatment table each day.	3.68 (0.96)	4.17 (0.95)	4.73 (0.53)	0.49 (<0.001*)	1.05 (<0.001*)
6	I am anxious about getting radiation treatment.	3.33 (1.09)	3.08 (1.18)	2.81 (1.33)	−0.25 (0.013*)	−0.52 (0.001*)
7	I am anxious about my cancer.	3.41 (1.00)	3.24 (1.10)	3.27 (1.08)	−0.17 (0.018*)	−0.15 (0.150)

Repeated measures pairwise t-tests showed statistically significant differences between the pre- and first post- questionnaires for all questions. When comparing between the pre- and finish-day post- questionnaires, all differences retained pairwise statistical significance except for question 7 regarding anxiety about their cancer.

Discussion

Our results indicate that this personalized tablet-based AR experience was helpful to patients. It decreased patient anxiety before starting treatment and helped improve understanding of how radiotherapy would treat their cancer.

This is consistent with what others have shown using similar immersive visualization technologies for patient education. Others have filmed 3D 360-degree videos for patient education showing what they will see when they enter the treatment vault every day for treatment, and these videos were found to be helpful [33], [34] although they do not show an “X-ray view” of the body or visible radiation beams.

VERT is a commercially available linac 3D simulator for training and education (VERTUAL, Yorkshire, U.K.). This type of 3D animation is viewed on a 2D projection screen while wearing 3D glasses. Originally developed for radiotherapy technologist training, the system has also been demonstrated to be helpful in patient education [22], [23], and VERT now also offers a library of patient education 3D videos for common cancer types. Both VERT and our smartphone AR system provide similar 3D experiences, depicting a realistic life-sized 3D animation with visible dynamic radiation beams targeting a translucent body, although neither is as immersive as a true head mounted display VR system [27]. Each system has its features and limitations. The professionally narrated VERT 3D videos have more comprehensive general background educational information for patients, but since they are pre-recorded, their system was not designed to be customized per patient.

A unique feature of our approach is that we showed each patient their own actual radiotherapy plan, not a generic plan. We found that patients became more fully engaged being aware that this demo was showing them a life-sized virtual rendition of themselves on the treatment table and could see how the radiation plan was customized for their cancer. The AR view allows patients to get up close and walk around the room to observe small details about their treatment from any viewing angle, and watching the virtual radiotherapy process often prompted patients to ask additional questions, such as the size/location of their cancer, or the proximity of adjacent organs at risk.

We previously reported on our original version of this system that uses a VR head-mounted display (HMD) [27]. The advantage of HMD systems is that they provide the most immersive and realistic experiences, but they require specialized hardware. Additionally, some patients have difficulty getting accustomed to the bulky HMD. The advantage of our tablet-based AR system is that no special hardware is required, and these AR visualizations are easier to use and can be appreciated by a broader population of patients. It can be viewed using the built-in browser on almost any modern smartphone (iOS or Android) without the need to download or install special software. Despite being not as immersive an experience, our study results indicate that patients still found this simpler 3D experience to be informative and helpful.

Allowing patients to easily download and view this on their own smartphone facilitates repeat viewing at home which may also help reinforce and sustain the educational and anxiety benefits over time. In our study, 83 % of patients loaded the AR view onto their own device. A large majority (79 %) watched it at least one more time at home, and 25 % said they actually watched it 4 or more times. Some participants commented that they appreciated the ability to show friends and family members this animation of their radiation treatment.

Prior studies have shown that subjects tend to retain new learned information better over time when presented in an immersive visual format [35], [36]. In comparing our immediately post-viewing surveys with end of treatment surveys, we found that most survey scores retained their improvement, suggesting that patients retained the educational benefits over time. One exception, however, is that we found that patients' decrease in anxiety about their cancer did not persist over time. Other researchers have also reported that not all benefits from VR remain persistent over time [33]. The cause of this effect requires further study, but one could hypothesize that if patients are experiencing acute RT toxicities on finish day (such as fatigue), this may have influenced perceptions of their cancer anxiety.

We wrote custom software that automates the process of taking standard DICOM-RT data from our commercial radiotherapy treatment planning system and converting it into the patient-specific AR animations, allowing us to easily conduct multiple educational sessions in one day. This conversion process is easily adaptable to other hardware platforms, including traditional dedicated VR/AR headsets, lower-cost smartphone-based “Google Cardboard” VR headsets, or even laptops or desktop computers.

This study has several limitations.

As a preliminary “proof-of-concept” study for this novel technology, we elected to have a single intervention group from one radiation oncology clinic and did not enroll a control group for comparison. Now that we have demonstrated the feasibility of this personalized patient education tool, our next step will be to conduct a larger randomized multi-institution study to assess the generalizability and effectiveness of this tool in broader settings.

Since most patients gradually become more accustomed to the daily routine of RT over the course of several weeks of treatment, we would expect that some of our outcome measures would tend to improve by the end of treatment, with or without this intervention. Based on our findings, one could consider that a benefit of this educational tool is that it can 'jump start' this effect, allowing patients to more quickly become comfortable with the process of radiotherapy prior to starting their treatment.

While the intended purpose of our intervention was to determine if it decreased patient anxiety, it is possible that the realism of this 3D visualization could have the opposite effect of increasing anxiety for some patients. Our survey items about anxiety (Q6, Q7) were intentionally phrased as neutral in order to detect this, and we did not see an increase at the time point immediately following the AR viewing. However, to be more certain to capture the possibility of this unintended effect, in the future, we may consider also including more specific post-viewing questions, e.g., “After seeing your tumor and the radiotherapy beams, are you now feeling more anxious than before the viewing?”

Rather than restricting enrollment to just one cancer type, we chose to enroll patients with any disease site, to test the feasibility of our system on a wide variety of radiotherapy plan types. With this heterogeneous group in this small pilot study, however, it may be difficult to draw generalizable conclusions about the benefit of our intervention in any one disease site or plan type.

Conclusions

In conclusion, we designed and built an application to render a 3D simulation of a patient's clinical radiotherapy treatment plan in augmented reality that can be viewed on a tablet or smartphone without additional hardware. This system can be used to enhance patient education for patients preparing to start radiotherapy. Our preliminary patient trial demonstrated that this tablet-based AR system improves patients' understanding of how radiotherapy will be used to treat their cancer and decreases their anxiety about undergoing radiotherapy treatment.

CRediT authorship contribution statement

Liam J. Wang: Conceptualization, Methodology, Software, Data curation, Formal analysis, Visualization, Writing – original draft, Writing – review & editing. Brian Casto: Software, Investigation, Data curation. Nancy Reyes-Molyneux: Investigation, Resources, Writing – review & editing. William W. Chance: Investigation, Resources, Writing – review & editing. Samuel J. Wang: Conceptualization, Methodology, Investigation, Writing – review & editing, Supervision.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.