Abstract
Objective
To evaluate the usability and adverse effects associated with virtual reality (VR) cognitive training and identify factors influencing them.
Design
Survey-based observational study.
Setting
Department of Rehabilitation Medicine in the hospital.
Participants
Twenty rehabilitation professionals (mean [standard deviation] age; 30.0[4.8] years, men 8[40%], and women 12[60%]) and 10 patients with stroke (mean [SD] age; 64.1[13.6] years, men 2[20%] and women 8[80%]).
Interventions
The participants wore a head-mounted display (Meta Quest2) and consecutively underwent 5 custom-designed cognitive training.
Main Outcome Measures
After the training, participants completed 3 questionnaires: the systemic usability scale, user experience questionnaire (UEQ), and cybersickness in VR questionnaire.
Results
The mean systemic usability scale score was 55.1 and 52.3 for rehabilitation professionals and patients, respectively. For the UEQ, the mean score for each item, including attractiveness, perspicuity, efficiency, dependability, stimulation, and novelty, were 0.9/0.2, 0.6/0.2, 0.5/−0.5, 1.2/0.8, 0.9/0.4, and 0.6/0.8 for rehabilitation professionals/patients, respectively. Rehabilitation professionals had slightly higher scores in most UEQ items. The mean cybersickness in VR questionnaire scores were 18.6 and 19.0 for rehabilitation professionals and patients, respectively.
Conclusions
Participants reported moderate usability and a generally below-average user experience, with mild-to-moderate VR sickness during VR cognitive training. The rehabilitation professionals rated usability higher than the patient group, while patients experienced more severe VR sickness. These findings may serve as a significant insight for developing VR cognitive training for application to patients in the future.
KEYWORDS: Cognitive training, Motion sickness, rehabilitation, usability, Virtual reality
Introduction
Stroke can induce various disabilities depending on the lesion location, including impairment of cognitive function.1 Cognitive impairment can be classified into mild cognitive impairment and dementia by severity, and 6%-27% of patients may develop dementia after a stroke.2 Cognitive impairment after a stroke affects various cognitive domains, such as memory, attention, executive function, and language.1 Furthermore, visuospatial neglect, where patients cannot perceive the opposite side of the damaged brain, is also frequently observed.3
Cognitive impairment is a crucial factor in functional independence and can significantly affect patients’ quality of life.4 Therefore, several medications that regulate various neurotransmitters are being used to treat cognitive impairment.5 Cognitive training is also actively used with medications, as it is known to improve specific cognitive domains through neural plasticity mechanisms.6
With the recent advancement of digital technology, computer-based cognitive training has been actively used. As mobile applications become more popular, various cognitive training programs using mobile applications are also being developed.7 Recently, cognitive training using virtual reality (VR) technology, which allows continuous interaction in environments closely resembling reality, is actively being developed.8 Through conventional or computer-based cognitive training, patients can receive training in various cognitive domains, but the limitation is that they cannot receive training for activities of daily living. Therefore, VR has the advantage of being able to enhance cognition through activities of daily living training by simulating actual daily environments in a virtual setting.9,10 Therefore, cognitive training using VR is expected to attract more attention and interest in the rehabilitation field.
However, when individuals wear a head-mounted display and experience VR, they may encounter VR sickness, which presents with symptoms similar to motion sickness, such as dizziness, nausea, headaches, and fatigue.11 One of the main reasons for VR sickness is the imbalance between visual and vestibular sensations in individuals.12 According to previous studies, up to a maximum of 80% of individuals may experience VR sickness.13 Individuals aged over 50 years or those who are not in their usual health condition are especially vulnerable.14,15 Studies on the occurrence of VR sickness during VR cognitive training have been conducted previously8,16; however, no quantitative research has evaluated its severity and analyzed vulnerability under specific conditions.
A lot of studies have indicated VR cognitive training improves the cognitive function of patients with stroke, and many of them focused on its clinical efficacy.17, 18, 19, 20 However, in the actual clinical setting, when attempting VR cognitive training with patients with cognitive impairments, many of them could not engage effectively in training and struggled with digital literacy.21 This study evaluated the usability and VR sickness rather than validating the clinical efficacy of VR cognitive training, allowing for an unbiased assessment.
In this study, we developed a cognitive training program using VR technology for various cognitive domains. We intended to quantitatively assess its usability, user experience, and the severity of VR sickness. Therefore, we aimed to examine factors influencing usability and VR sickness to provide a reference for the development of VR cognitive training in the future.
Methods
Participants
The study enrolled 30 participants, including 20 rehabilitation professionals (5 physiatrists and 15 occupational therapists) affiliated with the national insurance hospital and 10 patients with stroke. The patients with stroke were recruited from those admitted to or visited the hospital from September to December 2023. The inclusion criteria for patients with stroke were those aged ≥19 years, with a mini-mental state examination (MMSE) score ≥20. The exclusion criteria included a history of seizures and severe dizziness, visual or hearing impairments, problem with neck movement, difficulty maintaining a seated position, and the anticipation of severe VR sickness. Data on the age, sex, and VR experience of the study participants were collected. Similarly, information such as diagnosis, onset date, dominant hand, MMSE score, and modified Barthel index (MBI) was collected for the patients.
Contents of VR cognitive training
Two physiatrists (H.S. Kim and M.H. Bang) with over 10 years of clinical experience drew inspiration from existing computer cognitive training and developed 5 cognitive training contents focusing on memory, attention, executive function, and visuospatial function22 (fig 1). Moreover, the contents were made into software by Unipolar Inc Meta Quest 2 was used as the head-mounted display, and all designs and content were tailored to operate on Meta Quest 2. The training content was structured into levels based on difficulty. For each level, the training time and the number of correct and incorrect answers were stored as data to enable the comparison and evaluation of training outcomes. The above contents were developed for cognitive training purposes through research by physiatrists, but their efficacy has not yet been proven in specific patient populations.
Fig 1.
Representative scenes for the 5 cognitive training. (A) Making gimbap. (B) Finding luggage. (C) Sorting recyclables. (D) Picking red apples. (E) Finding subway exit.
Making gimbap
The training involves remembering the ingredients of the gimbap (Korean-style dried seaweed roll) initially presented, selecting the specific ingredients in sequence, and making the roll. By making the gimbap in the correct sequence, the training aims to improve memory and executive function. The training comprises a total of 9 levels, with the number of ingredients initially presented increasing as the levels progress.
Finding luggage
The training involves remembering and selecting the luggage initially presented among those rotating on the conveyor belt. By selecting the appropriate one from several similar luggage, the training aims to improve memory and attention. The training consists of 12 levels, with the number of luggage to be found increasing as the levels progress.
Sorting recyclables
The training involves correctly sorting each recyclable object into bins for cans, plastic, paper, and glass. By sorting recyclables into their appropriate categories, the training aims to improve executive function. The training consists of 8 levels, with the number of recyclables to be sorted increasing as the levels progress.
Picking red apples
The training involves picking apples from 20 apples hanging on 3 trees and placing them in the basket. This training aims to assess and educate individuals with hemispatial neglect or impairments in visuospatial perception. The degree of hemispatial neglect can be quantitatively assessed by considering the number and locations of the picked apples.
Finding subway exit
The training involves finding the designated subway exit (out of a total of 6) from where the individual has got off the subway. Visuospatial function and memory can be assessed and trained by remembering and finding the designated exit without errors.
Procedure
The participants wore a head-mounted display (Meta Quest 2) and consecutively underwent all possible levels of 5 cognitive training sessions for about 30 minutes (fig 2). Patients with stroke may experience a decrease in fine motor function of the dominant hand caused by muscle weakness, or the dominant hand may become the opposite of what it was before the stroke. Therefore, we enabled the VR controller to be held in the dominant hand after the stroke for training. After the training, 3 surveys were conducted on usability and VR sickness: the system usability scale (SUS), user experience questionnaire (UEQ), and cybersickness in virtual reality questionnaire (CSQ-VR).
Fig 2.
Scene of a rehabilitation professional wearing a head-mounted display and undergoing cognitive training.
Outcome measures
Scores were obtained for 3 survey results on usability, user experience, and VR sickness. We compared the scores between the rehabilitation professional and patient groups and also examined whether there were differences among the rehabilitation professionals based on their experience with VR.
System usability scale
The SUS is widely used to evaluate the usability of products and services, and it is a tool with proven reliability and validity.23,24 It comprises items that address whether the system is easy to use, technically consistent, and suitable for continued use in the future. The SUS consists of 10-item assessments. The item scores range from 1 (totally disagree) to 5 (totally agree) and are converted into a total score of 100 (Supplementary Table 1).
User experience questionnaire
The UEQ is a widely used and reliable measure of the subjective impression of the user's experience of the product.25 It consists of 26 items that are grouped into 6 scales (attractiveness, perspicuity, efficiency, dependability, stimulation, and novelty). Participants rate each item on a 7-point Likert scale. The answers are scaled from −3 (fully agree with the negative term) to +3 (fully agree with the positive term). The result is derived from the average of each item corresponding to the 6 scales, with a maximum score of 3 and a minimum of −3 (Supplementary Table 2).
Cybersickness in VR questionnaire
The CSQ-VR evaluates the whole range of cybersickness symptoms, including nausea, vestibular, and oculomotor symptoms. CSQ-VR is a valid measure of cybersickness and has superior psychometric properties compared to Simulator Sickness Questionnaire.26 There are 2 questions for each symptom type. Each question score ranges from 1 (absent feeling) to 7 (extreme feeling) (Supplementary Table 3). Each subscore of nausea, vestibular, and oculomotor symptoms is calculated as the sum of the 2 corresponding questions (2[questions]*1∼7[score range]=2∼14[subscore range]). The total score is the sum of the 3 subscores (2∼14[subscore range]*3=6∼42[total score range]).
Statistical analysis
The mean, SD, and minimum–maximum values of the continuous variables in demographic features were evaluated. Similarly, the mean and standard deviation of the survey results were derived, and the Mann–Whitney U test was used to compare the 2 groups. A P<.05 was considered statistically significant. SPSS Statistics 21.0 for Windows (IBM Corp)a was used for all analyses.
Ethics approval and consent to participate
This study was approved by the Institutional Review Board of the National Health Insurance Service Ilsan Hospital (IRB No. 2023-06-014-005) and conducted in accordance with the principles of the Declaration of Helsinki. All participants provided written informed consent.
Results
This study enrolled 30 participants, including 20 rehabilitation professionals (5 physiatrists and 15 occupational therapists) and 10 patients with stroke who had cognitive impairment. The mean age (SD) of the rehabilitation professionals and patients was 30.0 (4.8) years and 64.1 (13.6) years, respectively. Among the 20 rehabilitation professionals, 6 (30.0%) had experienced VR before the study, whereas 14 (70.0%) had no experience. None of the patients had prior experience with VR (table 1).
Table 1.
Demographic features of the rehabilitation professionals and patients
| Rehabilitation Professionals | Patients | ||
|---|---|---|---|
| Total | Number | 20 | 10 |
| Age (y) | Mean (SD) | 30.0 (4.8) | 64.1 (13.6) |
| Min-Max | 23-40 | 35-79 | |
| Sex | Men | 8 (40%) | 2 (20%) |
| Women | 12 (60%) | 8 (80%) | |
| VR experience | Yes | 6 (30%) | 0 (0%) |
| No | 14 (70%) | 10 (100%) |
The minimum age of patients participating in the study was 35 years, and the maximum was 79 years; 2 (20%) were men and 8 (80%) were women. The diagnoses included quadriparesis, right hemiparesis, and left hemiparesis in 2 (20%), 3 (30%), and 5 (50%) patients, respectively. The MMSE and MBI scores ranged from 23 to 30 and 26 to 81, respectively (table 2).
Table 2.
Patients baseline characteristics
| Patients | Sex | Age | Diagnosis | Onset | Dominant Hand | MMSE | MBI |
|---|---|---|---|---|---|---|---|
| 1 | F | 61 | Quadriparesis d/t SAH | 2017.03.24 | Rt | 30 | 73 |
| 2 | F | 55 | Lt. hemiparesis d/t Rt. MCA infarction | 2022.08.04 | Rt | 24 | 81 |
| 3 | F | 79 | Rt. hemiparesis d/t Lt. MCA infarction | 2022.12.31 | Lt | 23 | 58 |
| 4 | F | 79 | Lt. hemiparesis d/t Rt. BG infarction | 2023.09.25 | Rt | 26 | 56 |
| 5 | F | 35 | Lt. hemiparesis d/t Rt. MCA infarction | 2022.08.16 | Rt | 30 | 78 |
| 6 | M | 75 | Quadriparesis d/t Lt. pontine infarction | 2018.01.29 | Lt | 25 | 80 |
| 7 | M | 67 | Rt. hemiparesis d/t Lt. pontine ICH | 2023.05.12 | Lt | 27 | 53 |
| 8 | F | 59 | Lt. hemiparesis d/t Rt. BG ICH | 2005.12.01 | Rt | 27 | 73 |
| 9 | F | 58 | Rt. hemiparesis d/t Lt. BG ICH | 2012.12.01 | Lt | 25 | 65 |
| 10 | F | 73 | Lt. hemiparesis d/t Rt. ACA infarction | 2023.11.01 | Rt | 23 | 26 |
Abbreviations: Lt, left; Rt, right; MCA, middle cerebral artery; ACA, anterior cerebral artery; BG, basal ganglia; ICH, intracerebral hemorrhage;SAH, subarachnoid hemorrhage.
The mean SUS score (SD) for rehabilitation professionals was 55.1 (16.2), and for patients, it was 52.3 (19.2). Although the patient's mean score was slightly lower than that of rehabilitation professionals, the difference was not statistically significant.
For the UEQ, the mean score (SD) for each item, including attractiveness, perspicuity, efficiency, dependability, stimulation, and novelty, was 0.9 (1.3)/0.2 (1.7), 0.6 (1.4)/0.2 (2.1), 0.5 (1.1)/−0.5 (1.8), 1.2 (1.3)/0.8 (1.6), 0.9 (1.1)/0.4 (1.9), and 0.6 (1.1)/0.8 (1.4) for rehabilitation professionals/patients, respectively. Although scores for rehabilitation professionals and patients did not show statistically significant differences in each item, there was a slightly higher score for patients in novelty. In contrast, rehabilitation professionals had slightly higher scores in other items.
The mean CSQ-VR score (SD), which assesses the degree of VR sickness, was 18.6 (7.8) for rehabilitation professionals and 19 (12.9) for the patients. Regarding the 3 subscores, the vestibular score was higher for patients, and the oculomotor score was higher for rehabilitation professionals; however, the difference between the 2 groups was not statistically significant (table 3).
Table 3.
Comparison of scores between rehabilitation professionals and patients
| Scale | Subscale | Rehabilitation Professionals (N=20) | Patients (N=10) | P Value |
|---|---|---|---|---|
| SUS | Mean (SD) | 55.1 (16.2) | 52.3 (19.2) | .914 |
| UEQ | Attractiveness | 0.9 (1.3) | 0.2 (1.7) | .350 |
| Perspicuity | 0.6 (1.4) | 0.2 (2.1) | .983 | |
| Efficiency | 0.5 (1.1) | −0.5 (1.8) | .169 | |
| Dependability | 1.2 (1.3) | 0.8 (1.6) | .650 | |
| Stimulation | 0.9 (1.1) | 0.4 (1.9) | .619 | |
| Novelty | 0.6 (1.1) | 0.8 (1.4) | .914 | |
| CSQ-VR | Nausea | 7.2 (3.2) | 7.2 (4.3) | .948 |
| Vestibular | 4.8 (2.8) | 6.4 (4.7) | .559 | |
| Oculomotor | 6.7 (3.2) | 5.4 (4.8) | .198 | |
| Total | 18.6 (7.8) | 19.0 (12.9) | .713 |
We compared the subgroup of rehabilitation professionals who had prior VR experience (N=6) with those who had no prior experience (N=14) to examine if there were differences in each item. For the SUS score, the group with prior experience had a mean (SD) of 57.9 (13.2), whereas the group without prior experience had a mean (SD) of 53.9 (17.6). Although the group with experience had a higher average, the difference was not statistically significant. For UEQ scores, the group with experience had higher averages in attractiveness and novelty, whereas the group without experience had higher averages in perspicuity, efficiency, dependability, and stimulation. However, none of these differences showed statistical significance. For CSQ-VR scores, the group with experience had a mean (SD) of 15.5 (5.5), whereas the group without experience had a mean (SD) of 19.9 (8.4). Although the group with experience had a slightly lower average, this difference was not statistically significant (table 4).
Table 4.
Comparison of scores based on rehabilitation professionals’ virtual reality experience
| Scale | Subscale | VR Experience |
||
|---|---|---|---|---|
| Yes (N=6) | No (N=14) | P Value | ||
| SUS | Mean (SD) | 57.9 (13.2) | 53.9 (17.6) | .718 |
| UEQ | Attractiveness | 1.2 (1.1) | 0.7 (1.3) | .602 |
| Perspicuity | 0.2 (0.9) | 0.7 (1.5) | .494 | |
| Efficiency | 0.3 (1.3) | 0.6 (1.1) | .841 | |
| Dependability | 1.0 (1.4) | 1.3 (1.2) | .718 | |
| Stimulation | 0.8 (1.5) | 0.9 (1.0) | .602 | |
| Novelty | 0.8 (1.4) | 0.5 (1.0) | .547 | |
| CSQ | Nausea | 5.3 (2.2) | 7.9 (3.4) | .109 |
| Vestibular | 4.5 (2.5) | 4.9 (3.0) | .904 | |
| Oculomotor | 5.7 (2.6) | 7.1 (3.4) | .353 | |
| Total | 15.5 (5.5) | 19.9 (8.4) | .353 | |
Discussion
In this study, we developed 5 cognitive training programs using VR technology, applied them to participants, and quantitatively evaluated usability, user experience, and the degree of VR sickness through surveys. Participants reported moderate usability and below-average user experience in general and also complained of mild-to-moderate VR sickness when undergoing VR cognitive training. These findings may be valuable references for consideration when developing and applying VR cognitive training to patients.
Several studies suggest VR cognitive training that improves the cognitive function of patients with mild cognitive impairment or dementia.8,17 However, when attempting computer-based or VR cognitive training with patients in the hospital, many individuals express resistance to using computers or VR technology, leading to challenges in active training. This could be attributed to the fact that most patients encountering VR technology for the first time, especially patients with stroke who tend to be older, may have a limited understanding of digital technology.21 As seen in our preliminary study analyzing mobile applications for cognitive training, the most crucial aspect of cognitive training is the patient's direct engagement; in other words, how well they can use it.7 Unlike other one-time treatments, prolonged engagement in cognitive training ensures efficacy.27 Therefore, we researched the usability and user experience aspects to understand which individuals can effectively use VR cognitive training.
When comparing the SUS scores between the rehabilitation professional and patient groups, rehabilitation professionals reported slightly higher usability. However, both scores fell within the “okay” range on the adjective rating of the SUS scores, indicating moderate usability.28 Additionally, when evaluating the user experience with the UEQ, rehabilitation professionals showed higher scores than the patients in all the items, excluding novelty. Regarding the subscores, rehabilitation professionals scored above average only in dependability, whereas patients scored above average only in novelty. Both groups reported below-average to bad user experience in all other subscores.29 The values reported in our study are slightly lower compared to those of previous studies, which indicated the above-average usability for VR among older people.30 This could be attributed to the fact that while other studies focused on physical training purposes such as motor or balance improvement, our study majored in cognitive enhancement, which might have been perceived as slightly more challenging.31,32
The higher usability and user experience observed mainly among rehabilitation professionals can be attributed to the older average age of the patients,33 suggesting a lower level of digital literacy in understanding and using digital technology. In addition, cognitive impairments and paralytic weakness among patients may have negatively affected their digital literacy.34 Because there is still insufficient evidence regarding the factors influencing the literacy of VR and other digital technology, further research is needed to explore the factors affecting digital literacy. In contrast, when comparing the rehabilitation professional group with VR experience to those without, the SUS score, attractiveness, and novelty for UEQ scores were higher for the group with experience. Individuals with prior experience with VR might be more interested in and positively disposed to new technologies like VR.
VR sickness refers to symptoms such as nausea, vomiting, and headaches that individuals may experience when exposed to VR environments.35 VR sickness is known to occur because of an imbalance between the visual and vestibular senses.12 This symptom manifests differently in individuals, with those aged >50 years and individuals not in their usual stage of health being more vulnerable.14,15 In this study, the CSQ-VR scores, which assess the severity of VR sickness, suggest mild-to-moderate VR sickness were reported. Despite the fact that most participants experienced VR for the first time, not many complaints of sickness were reported. The slightly higher CSQ-VR scores in patients may be attributed to the significantly higher average age in the patient group. In addition, likely, old age and the health condition affected by the stroke accounted for this difference.15
Furthermore, the group with previous experience with VR showed less VR sickness in specific items such as nausea, vestibular, and oculomotor compared to the group without such experience. Because VR sickness is known to be severe when experiencing VR for the first time and gradually declines as individuals adapt, the group with previous experience in VR experiences less sickness.36 However, this study did not show statistically significant differences, possibly because of the small sample size of the study participants and the considerable variability of VR motion sickness among individuals. Therefore, in the future, it will be essential to conduct in-depth research on the characteristics of individuals vulnerable to VR sickness, considering ways to minimize the sickness.
Study limitations
First, the most significant limitation of this study is that it did not examine clinical efficacy in the patients. Evaluating efficacy and safety is crucial for adopting and using new digital technology like VR. However, unlike previous research that evaluated the efficacy, the uniqueness of this study lies in the quantitative assessment of usability and VR sickness.
Second, the number of patients participating in the study was small, and we could not collect information on various variables that could affect usability and VR sickness. Therefore, we could not conduct an in-depth analysis regarding which characteristics of patients influence usability and VR sickness. A small sample size may lead to selection bias, making it difficult to derive and justify inferential significance from the results of this study. Future research should be conducted on a more significant number of patients to determine which characteristics contribute to higher usability and VR sickness.
Finally, it is challenging to consider the results reported by rehabilitation professionals and patients as representative of the entire VR cognitive training. This study conducted experiments and evaluations only using a software program developed by one research group. However, through this study, we could identify which aspects to consider when designing VR cognitive training in the future.
Conclusions
We developed 5 new cognitive training programs using VR technology and quantitatively evaluated usability, user experience, and the degree of VR sickness through surveys. Both rehabilitation professionals and patients reported moderate usability and poor user experience, as well as average or below levels of VR sickness. Usability was higher in the rehabilitation professional group than in the patient group, and VR sickness was more severe in the patient group. Moreover, patients have physical disabilities and experience VR for the first time, which may lead to reported severe sickness. These findings could be essential references for developing cognitive training using VR technology and applying it to patients in the future.
Suppliers
a. SPSS Statistics 21.0 for Windows; IBM Corp.
Disclosure
The authors declare that they have no competing interests.
Acknowledgments
Acknowledgments
The authors would like to thank all the participants who collaborated in this study.
Authorship contributions
MHB (Myeonghwan Bang), SSK (Sung Shin Kim), and HSK (Hyoung Seop Kim) were responsible for the conception and design of the study. MHB and MAK (Min A Kim) collected data, and MHB performed data analyses. MHB drafted the manuscript. MHB, SSK, and HSK contributed to the interpretation of the results and critical revision of the manuscript for important intellectual content, and they approved the final version of the manuscript. All the authors have read and approved the final manuscript.
Data statements
The data sets used and/or analyzed during the current study are available from the corresponding author upon reasonable request.
Footnotes
This study was supported by the Rehabilitation Research and Development Support Program (#NRCRSP-EX22009) of the National Rehabilitation Center, Ministry of Health and Welfare, Korea.
Supplementary material associated with this article can be found, in the online version, at doi:10.1016/j.arrct.2024.100378.
Appendix. Supplementary materials
References
- 1.Rost NS, Brodtmann A, Pase MP, et al. Post-stroke cognitive impairment and dementia. Circulation Res. 2022;130:1252–1271. doi: 10.1161/CIRCRESAHA.122.319951. [DOI] [PubMed] [Google Scholar]
- 2.Sun JH, Tan L, Yu JT. Post-stroke cognitive impairment: epidemiology, mechanisms and management. Ann Transl Med. 2014;2:80. doi: 10.3978/j.issn.2305-5839.2014.08.05. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Esposito E, Shekhtman G, Chen P. Prevalence of spatial neglect post-stroke: a systematic review. Ann Phys Rehabil Med. 2021;64 doi: 10.1016/j.rehab.2020.10.010. [DOI] [PubMed] [Google Scholar]
- 4.Oros RI, Popescu CA, Iova CA, Mihancea P, Iova SO. The impact of cognitive impairment after stroke on activities of daily living. Hum Vet Med 2016;8:41-44.
- 5.Baskys A, Cheng JX. Pharmacological prevention and treatment of vascular dementia: approaches and perspectives. Exp Gerontol. 2012;47:887–891. doi: 10.1016/j.exger.2012.07.002. [DOI] [PubMed] [Google Scholar]
- 6.Leung NT, Tam HM, Chu LW, et al. Neural plastic effects of cognitive training on aging brain. Neural Plast. 2015;2015 doi: 10.1155/2015/535618. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Bang M, Jang CW, Kim HS, Park JH, Cho HE. Mobile applications for cognitive training: content analysis and quality review. Internet Interv. 2023;33 doi: 10.1016/j.invent.2023.100632. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Zhong D, Chen L, Feng Y, et al. Effects of virtual reality cognitive training in individuals with mild cognitive impairment: a systematic review and meta-analysis. Int J Geriatr Psychiatry. 2021;36:1829–1847. doi: 10.1002/gps.5603. [DOI] [PubMed] [Google Scholar]
- 9.Buele J, Varela-Aldás JL, Palacios-Navarro G. Virtual reality applications based on instrumental activities of daily living (iADLs) for cognitive intervention in older adults: a systematic review. J Neuroeng Rehabil. 2023;20:168. doi: 10.1186/s12984-023-01292-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Faria AL, Andrade A, Soares L. Benefits of virtual reality based cognitive rehabilitation through simulated activities of daily living: a randomized controlled trial with stroke patients. J Neuroeng Rehabil. 2016;13:1–12. doi: 10.1186/s12984-016-0204-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Chang E, Kim HT, Yoo B. Virtual reality sickness: a review of causes and measurements. Int J Hum Comput Interact. 2020;36:1658–1682. [Google Scholar]
- 12.Martirosov S, Kopecek P. Cyber sickness in virtual reality–literature review. Annals DAAAM & Proc. 2017;28 [Google Scholar]
- 13.Oh H, Son W. Cybersickness and its severity arising from virtual reality content: a comprehensive study. Sensors. 2022;22:1314. doi: 10.3390/s22041314. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Brooks JO, Goodenough RR, Crisler MC, et al. Simulator sickness during driving simulation studies. Accid Anal Prev. 2010;42:788–796. doi: 10.1016/j.aap.2009.04.013. [DOI] [PubMed] [Google Scholar]
- 15.Kolasinski EM. Simulator sickness in virtual environments. United States: Army Research Institute for the Behavioral and Social Sciences 1995.
- 16.Park JH, Liao Y, Kim DR, et al. Feasibility and tolerability of a culture-based virtual reality (VR) training program in patients with mild cognitive impairment: a randomized controlled pilot study. Int J Environ Res Public Health. 2020;17:3030. doi: 10.3390/ijerph17093030. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Maggio MG, Latella D, Maresca G, et al. Virtual reality and cognitive rehabilitation in people with stroke: an overview. J Neurosci Nurs. 2019;51:101–105. doi: 10.1097/JNN.0000000000000423. [DOI] [PubMed] [Google Scholar]
- 18.Maggio MG, Maresca G, De Luca R, et al. The growing use of virtual reality in cognitive rehabilitation: fact, fake or vision? A scoping review. J Natl Med Assoc. 2019;111:457–463. doi: 10.1016/j.jnma.2019.01.003. [DOI] [PubMed] [Google Scholar]
- 19.Masoumzadeh S, Moussavi Z. Does practicing with a virtual reality driving simulator improve spatial cognition in older adults? A pilot study. Neurosci Insights. 2020;15 doi: 10.1177/2633105520967930. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Zhu K, Zhang Q, He B, Huang M, Lin R, Li H. Immersive virtual reality-based cognitive intervention for the improvement of cognitive function, depression, and perceived stress in older adults with mild cognitive impairment and mild dementia: pilot pre-post study. JMIR Serious Games. 2022;10:e32117. doi: 10.2196/32117. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Susło R, Paplicki M, Dopierała K, Drobnik J. Fostering digital literacy in the elderly as a means to secure their health needs and human rights in the reality of the twenty-first century. Fam Med Prim Care Rev. 2018;20:271–275. [Google Scholar]
- 22.Kueider AM, Parisi JM, Gross AL, Rebok GW. Computerized cognitive training with older adults: a systematic review. PLOS ONE. 2012;7:e40588. doi: 10.1371/journal.pone.0040588. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Brooke J. SUS-A quick and dirty usability scale. Usability Eval Ind. 1996;189:4–7. [Google Scholar]
- 24.Klug B. An overview of the system usability scale in library website and system usability testing. Weave J Libr User Exp. 2017;1 [Google Scholar]
- 25.Laugwitz B, Held T, Schrepp M. HCI and Usability for Education and Work: 4th Symposium of the Workgroup Human-Computer Interaction and Usability Engineering of the Austrian Computer Society, USAB 2008, Graz, Austria, November 20-21, 2008. Proceedings 4. Springer; 2008. Construction and evaluation of a user experience questionnaire; pp. 63–76. [Google Scholar]
- 26.Kourtesis P, Linnell J, Amir R, Argelaguet F, MacPherson SE. Cybersickness in Virtual Reality Questionnaire (CSQ-VR): a validation and comparison against SSQ and VRSQ. Virtual Worlds. 2023;2:16–35. [Google Scholar]
- 27.Lumsden J, Edwards EA, Lawrence NS, Coyle D, Munafò MR. Gamification of cognitive assessment and cognitive training: a systematic review of applications and efficacy. JMIR Serious Games. 2016;4:e5888. doi: 10.2196/games.5888. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Bangor A, Kortum P, Miller J. Determining what individual SUS scores mean: adding an adjective rating scale. J Usability Stud. 2009;4:114–123. [Google Scholar]
- 29.Schrepp M. User experience questionnaire handbook. All you need to know to apply the UEQ successfully in your project. 2015. https://www.ueq-online.org/Material/Handbook.pdf.
- 30.Tuena C, Pedroli E, Trimarchi PD, et al. Usability issues of clinical and research applications of virtual reality in older people: a systematic review. Front Hum Neurosci. 2020;14:93. doi: 10.3389/fnhum.2020.00093. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Pedroli E, Greci L, Colombo D, et al. Characteristics, usability, and users experience of a system combining cognitive and physical therapy in a virtual environment: positive bike. Sensors. 2018;18:2343. doi: 10.3390/s18072343. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Wüest S, Borghese NA, Pirovano M, Mainetti R, van de Langenberg R, de Bruin ED. Usability and effects of an exergame-based balance training program. Games Health J. 2014;3:106–114. doi: 10.1089/g4h.2013.0093. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Oh SS, Kim KA, Kim M, Oh J, Chu SH, Choi J. Measurement of digital literacy among older adults: systematic review. J Med Internet Res. 2021;23:e26145. doi: 10.2196/26145. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Park EY, Nam SJ. An analysis of the digital literacy of people with disabilities in Korea: verification of a moderating effect of gender, education and age. Int J Consum Stud. 2014;38:404–411. [Google Scholar]
- 35.LaViola JJ., Jr. A discussion of cybersickness in virtual environments. ACM Sigchi Bulletin. 2000;32:47–56. [Google Scholar]
- 36.Palmisano S, Constable R. Reductions in sickness with repeated exposure to HMD-based virtual reality appear to be game-specific. Virtual Real. 2022;26:1373–1389. [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.