An immersive virtual reality-based application for treating ADHD: A remote evaluation of acceptance and usability

Abstract

Introduction

Virtual reality (VR) is a digital technology currently considered to implement rehabilitation programs for children with ADHD, a disorder characterised by inattention, overactivity and impulsiveness. This study presents the results of the acceptance and usability of a VR application developed for children with ADHD aiming to provide an environment capable of supporting the development of the different attentional components. Due to COVID-19 restrictions, this study had the secondary aim of assessing whether a remote evaluation was feasible and meaningful.

Methods

A sample of 20 clinical experts (neuro and psychomotor therapists of the developmental age) was involved in assessing the proposed environment. Two different tools have been applied: the Technology Acceptance Model (TAM-3) questionnaire and a semi-structured interview were self-administered. Six sessions were planned in total, and each one lasted 30 min.

Results

With respect to the acceptance of the system, the mean of the answers given is for most of the constructs greater than 4, showing agreement among experts. Cronbach alpha and correlations of subscales seem to confirm the reliability of measures. According to results from the interviews, the developed application has shown versatility in being able to be applied to the heterogeneity of the disorder and it was also possible to obtain valuable insights on possible additional features and functionalities. Regarding the secondary aim, the collected outcomes were positive: all the participants were satisfied with what they could perceive about the application.

Conclusions

The results of this work pave the way for a future validation study with children due to the active participation of clinicians and their unanimous positive judgement confirming that the application was considered user-friendly and well accepted.

Introduction

Attention deficit hyperactivity disorder (ADHD) is a disorder that is usually diagnosed during school years and is characterised by inattention, overactivity and impulsiveness.¹ Symptoms can characterise ADHD differently depending on the specific case considered. Still, they are all associated, in the long term, with an increased risk of educational failure, developing intrapersonal issues and mental illness. Currently, ADHD treatment varies according to the disorder's severity. Sometimes rehabilitation and psychological programs are associated with a pharmacological intervention to mitigate the impulsive and hyperactive aspects.^2,3

Virtual reality (VR) is a digital technology that has been applied to different medical areas such as neuro-motor and cognitive rehabilitation.^4–6 Given the different studies demonstrating its benefits, VR is also considered to implement rehabilitation programs for children with ADHD.⁷ In fact, it has a greater ecological validity; it is possible to quantify measures that would not be possible to consider through traditional tests; it allows to perform behavioural monitoring that is recorded in the VR system; the environment created can be customised based on the characteristics of the patient.^8,9 Moreover, it can be designed to provide fun ad engagement, motivating the patient to use this new technology.¹⁰ In addition, VR can turn an inactive learning experience into an active performance.¹¹ It could be considered as an ‘embodied technology’ that provides the sensation of presence, allowing interaction with virtual objects in the 3D space. Generating simulations of one's own body in the world can promote self-regulation, motivation and learning, and the quality of experience can improve the movements, actions and emotions, adapting them to the context.⁷

Several studies have already attempted to use digital and VR technologies as an instrument to help children with ADHD in learning to better manage their symptoms, improving the cognitive performance of children and adolescents, especially in vigilance and attention tasks, but also in the control of inhibitory and impulsivity in the general behaviour and learning abilities.^3,12–14 Given this, many professionals, doctors and therapists are interested in adopting novel digital technologies,¹⁵ and the FDA has approved a mobile app as a therapy for ADHD,¹³ underlining the increase of attention towards this field.

Regarding VR, its most known application in ADHD treatment is the Virtual Classroom (VC),⁸ which has been shown to represent valuable support for both ADHD evaluation and as a promising rehabilitation tool of attentional processes, measuring commission and omission errors, average response time, motor activity and quality of attention.^16–20

Despite this growing attention, recent literature reviews highlighted that, besides many proposals, only a few digital applications have been adequately assessed in scientifically sound studies. For instance, the review performed by Păsărelu et al.²¹ investigated the effectiveness of mobile-based apps for ADHD, found 51 studies, but only one met the inclusion criteria. Therefore, the authors concluded that although promising, digital technologies still need to be fully validated in this population. The same conclusion was reached in another review addressing the potential of emerging technologies to manage ADHD.¹⁴ In this work, Cibrian et al. also highlighted the need to focus on designing VR applications, properly assessing the accessibility of these technologies, which otherwise may be unfeasible and uncomfortable for children.

Specifically, regarding immersive VR, Romero-Ayuso et al.⁷ performed a meta-analysis including four studies; their results demonstrated that VR interventions could help increase sustained attention in children with ADHD, improving correct hits, reaction times and perceptual sensitivity while reducing omissions. Nonetheless, the same review also highlighted that almost all studies used VC and that the focus was only on cognitive tasks, whereas including self-regulation and working memory exercises would also have been helpful. Another critical point raised by such a review is related to the lack of transfer of the acquired benefits to daily life. Thus, the increase of ecological validity – together with the choice of an adequate age-related level of challenge – is an objective that still has to be fulfilled.

Given the current state-of-the-art, we have designed and developed an immersive VR-based serious game for children with ADHD with the aim of (a) promoting orientation of attention; (b) inhibiting impulsiveness; (c) increasing sustained attention spans; (d) strengthening memory and (e) motivate the children.²² Such a serious game exploits immersive VR and differentiates from previously developed ones because it provides higher interaction levels and a different context. Indeed, the ADAD VR experience takes place in a living room and allows the children to walk around freely and interact with all the virtual objects in the scene with no constraints (Figure 1). We expect these two elements to improve the quality of the proposed intervention because, first, the living room represents a familiar environment, which may enhance the ecological validity of the scenario, promoting the transfer of acquired abilities to the activities of daily living. Second, we hypothesise such an environment to be more motivating and engaging than previous solutions because of the higher control (also called sense of agency) that the child could exert over the environment. Literature studies investigating sense of presence and motivation have indeed demonstrated how this factor positively influences both.^23–25 The primary aim of the present study was to evaluate the acceptance and usability of ADAD to assess how it can represent a meaningful and motivating environment for children with ADHD, capable of supporting the development of the various attentional components and offering an engaging experience. To determine whether such an application could be feasible and appropriate, especially in terms of the provided challenge for children of different ages,⁷ we first involved ADHD therapists in the role of experts.

Figure 1.

The virtual environment of the application (top), and two screenshots while interacting with the objects (bottom).

Our first intent was to perform a usability study making therapists try the application. However, due to COVID-19 pandemic, this was not possible. Therefore, we decided to perform such an evaluation remotely. This choice was supported by successful remote experiences in other fields, for example, leisure²⁶ and learning.²⁷

Given this, this study had the secondary aim of assessing whether a remote evaluation of a medical-oriented VR application – performed using a commercial teleconferencing platform – was feasible and meaningful.

Methods

The ADAD application was developed using Unity, a cross-platform game engine developed by Unity Technologies.²⁸ The VR system chosen to visualize and interact with the VR environment is the Oculus Quest 2 by Meta.²⁹ The serious game focuses on the activity of daily living in a room setting furnished with few elements. It is characterised by a simple game dynamic, able to be understood by school-age children, that consists in finding a target object, shown in front of a blackboard for a specific time, grabbing and placing it in a basket, according to the instructions given by a digital voice. The game is composed of six levels with increasing difficulty in terms of target objects, target object viewing time, time to complete the task, visual and audio distractors. Visual aids and vocal reinforcement are also provided. Further details are described in Bernardelli et al.²²

Participants and recruitment

To assess the acceptance and usability of the proposed environment as a tool for supporting children suffering from ADHD, we involved experts identified in clinical neuro and psychomotor therapists of the developmental age.

Before, we had also involved bioengineers (technicians group) with a background in VR and experience with the technology. Including this type of expert has allowed us to further test the application's functionality and design; the results of such validation were reported in a previous paper.¹³

This study was conducted in January 2021 and the protocol was approved by the University of Milan Ethical Committee (n. 77/21, date 13/07/2021). Participants were recruited from the authors’ social and professional networks and asked to participate voluntarily in the study after signing informed consent. A total of 39 clinicians were contacted; 31 gave informed consent and 20 of them attended the meetings. A total of 18 of the participants from the clinicians’ groups were female (90%) and 2 were male (10%). The average age is 30.25 years. Table 1 shows detailed participants’ socio-demographics characteristics.

Table 1.

Socio-demographic characteristics of the therapists participating in ADAD remote assessment.

Characteristics	Participants (n = 20)
	N	%
Age range (years)
24–29	11	55
30–34	7	35
35–39	1	5
40–45	-
>45	1	5
Sex
Male	2	10
Female	18	90
Work experience (years)
1–5	11	55
6–10	6	30
11–15	2	10
16–20	-
>20	1	5
VR device already tested	6	30%

Protocol

Due to the COVID-19 pandemic, which dramatically restricted face-to-face activities, the evaluation has been performed remotely through the Skype platform. Participants have been thus asked to attend a session on the VR-based serious game and to answer the questionnaires presented in the Outcome section. The meetings were organised in separate small groups (3–4 participants), depending on the availability of the participants detected using a Doodle. Six sessions were planned in total in December 2020, and each one lasted approximately 30 min. After an initial introduction illustrating the dynamics of the game and the sequence of related levels, each session was focused on the vision of a game session performed online by a researcher of the team. Possible questions were answered at the end of the demo, and evaluation materials were distributed by email after the session.

Outcomes

For the study, we have applied two different tools: the first one is the Technology Acceptance Model (TAM-3) questionnaire,^30,31 which represents a predictive statistical model of the acceptance of technology for its future application.³¹ It has been widely used in literature to analyse the acceptance and use of information technologies, such as VR, in different contexts^32,33 and evaluate the acceptance of disposal by patients in rehabilitation contexts.^34,35

In particular, the following 8 constructs were used for a total of 27 items answered on a 7-point Likert scale (from ‘Strongly Disagree’ to ‘Strongly Agree’):

• - perceived usefulness, 4 items (PU);

• - perceived ease of use, 4 items (PEAU);

• - perceived enjoyment, 3 items (ENJ);

• - image, 3 items (IMG);

• - job relevance, 3 items (REL);

• - output quality, 3 items (OUT);

• - result demonstrability, 4 items (RES);

• - behavioural intention, 3 items (BI).

Besides the TAM questionnaire, a semi-structured interview was also self-administrated to participants in order to investigate some more specific aspects of the serious game as, for example, the realism of the VE, the length of the game, difficulty levels and their progression, hints and guides, strengths and weaknesses and possible suggestions for improvements. Materials for the semi-structured interview have been validated internally through a focus group of experts.

Data analysis

Data analysis took place after collecting questionnaires from experts by email in January 2021. For what concerns the TAM questionnaire, statistical analysis was performed using MATLAB 2020a Statistical Toolbox. Cronbach's alpha was used to assess the consistency of TAM3 subscales, as some of the TAM3 items were not included.³⁰ Pearson's correlations were used to determine the correlation between the investigated subscales. Referring to the open questions of the semi-structured interview, answers collected have been analysed and coded according to a thematic analysis performed by two separate researchers to guide further improvements of the application.

Results

Technology acceptance model questionnaire

With respect to the acceptance of the system, descriptive statistics of the subscales related to the responses of the 20 clinicians are reported in Table 2. Cronbach alpha was mostly very high for all the subscales, thus indicating their reliability; RES subscale obtained a lower score, but it was considered acceptable (α > 0.6) and subscales’ correlations seem to confirm the measures’ reliability.

Table 2.

TAM results from expert evaluation for the eight variables related to the acceptability of the system.

	Mean	SD	Cronbach alpha
PU	4.75	0.72	0.86
PEOU	5.21	0.49	0.72
ENJ	5.97	0.15	0.83
IMG	3.53	1.12	0.73
REL	4.58	0.75	0.86
OUT	5.02	0.23	0.76
RES	5.43	0.29	0.66
BI	5.12	1.27	0.81

The mean of the answers given is for most of the constructs greater than 4; the participants assigned a score mainly between ‘neutral’ and ‘strongly agree’ to the assessed items. Only in the IMG (image) construct value of 3 is observed, showing a disagreement among therapists (also, the standard deviation is higher). About the evaluation of the PU (perceived usefulness) construct, 64% of participants believe that the proposed digital environment can be useful in the rehabilitation setting and also allows to increase professional productivity.

Moreover, 100% of clinicians found the VR tool enjoyable and fun and 71% of participants in the evaluation of the behavioural intention (BI) construct reported that if they had access to the VR tool, they would use it.

Table 3 reports the correlations among subscales.

Table 3.

Correlations between TAM-3 subscales.

	PU	PEOU	ENJ	IMG	REL	OUT	RES	BI
PU	-
PEOU	0.28
ENJ	0.52*	0.19
IMG	0.09	0.43	−0.32
REL	0.69**	0.06	0.03	0.15
OUT	0.47*	0.42	−0.05	0.49*	0.67*
RES	0.47*	0.56*	0.55*	0.12	0.15	0.26
BI	0.43	−0.12	0.26	−0.18	0.18	0.02	0.00	-

Semi-structured interview

Data analysis in Figure 2 showed that all participants have had professional experience with children who presented attention deficit with both primary and secondary disorders; 55% work with both types of patients, while 40% with children with the primary or secondary disorder.

Figure 2.

Results (frequencies) of the semi-structured interview self-administered after the session. Numbers represent the percentage of respondents.

Among them, 55% use IT or technological material, such as a PC or a tablet, based on the specific therapeutic objectives of the child being treated and his/her needs. On the other hand, 45% of clinicians prefer not to use digital tools, both for the scarce technological resources available in the structure where they work and because for the type of patients being treated, among the methodologies of the intervention, movement games or other creative activities were preferred (Table 3).

About 94% of the interviewed experts believe that the conceived digital environment could constitute a possible future resource for them to be used in a work context due to the growing interest and use of technologies such as VR in the therapeutic field. Only 5% of them think that, even if it could represent a resource, it would be difficult to use it in a rehabilitation setting. Those who agreed also affirm that this technology enables greater control and better management of distracting stimuli.

Regarding whether the proposed digital environment was more suitable for rehabilitation or assessment purposes, 55% of respondents believed that it is more appropriate as a rehabilitation tool in a therapeutic setting. However, 25% of participants think that it can be contextualised in both areas since, thanks to the possibility of recording the data relating to the sessions, it can catch elements of the evolution of the child's skills (response times, type of errors, etc.). Due to the kind of scenario and requests, the application can be a useful rehabilitation tool, motivating the child and helpful in learning specific skills. Finally, 20% of clinicians believed that the application could represent an evaluation tool exclusively. In their opinion, only if there is the possibility to configure the environment and the difficulty level for the specific child could the game also be considered for rehabilitation purposes.

For what concerns the age group in which to propose the VR application, 50% of clinicians agreed to suggest it from 6 years of age on. About 15% believe it is suitable from 7 years of age; 10% from 8 years of age and 5% only after 11 years of age. One clinician affirms that it is necessary to evaluate not particularly the age of the child but instead his/her clinical profile, considering, consequently, the use of the environment, also in relation to the specific clinical characteristics.

Finally, 95% of clinicians answered affirmatively to the question investigating the feasibility of the remote validation, thanks to the fact that the tasks performed by the operator during the game session were clear. Only 5% answered ‘enough’. Furthermore, 90% of participants would have been able to complete the required tasks independently.

The questionnaire also asked to identify the strengths and criticalities of the virtual environment and suggest possible improvements (Figure 3). These comments are categorised into different topics according to related frequencies, as reported in Figure 4.

Figure 3.

The evaluation of questions 8–13 (means and SD) was based on a 5-points Likert scale (1–5).

Figure 4.

Results (frequencies) in terms of strengths and criticalities of the VR application emerged from the analysis of comments of the semi-structured interview.

Concerning the strengths of the virtual environment, 75% of respondents believe that it favours the activation and motivation of the child, with the consequent investment of his cognitive resources and 55% appreciate the gradual increase in the difficulty of the different levels of the application. With regard to criticisms, 50% of clinicians believe that the guiding digital voice, while understandable, is unnatural and uncomfortable. Moreover, 20% of them think that the fact that it is not possible to configure the virtual environment and select the degree of difficulty, depending on the child being treated, is a limit.

Discussion

This work presents an expert evaluation, performed by neuro- and psychomotor therapists, of an immersive VR-based application dedicated to children with ADHD. ADAD was developed to target several attention domains and improve both the ecological validity of the treatment and the children's engagement by providing them with a more interactive experience. With the present study, we proved the acceptance and usability of the ADAD application and the feasibility of an expert evaluation via a teleconferencing system.

Regarding our primary aim, the results were positive and encouraging as they showed great interest in the medical personnel involved. Considering that all the participants of the study have had previous yearly experience with children with attention difficulty, we can argue that they represented a valid sample in evaluating the characteristics of the environment.

Given the results of both the TAM questionnaire and the semi-structured interview, the therapists evaluated the application with positive feedback, considering it a possible resource to be used in a future therapeutic setting. Moreover, therapists reported positive comments regarding the versatility of ADAD, specifying that it could be helpful for children with the heterogeneous manifestation of the disease.³⁶

It should also be observed that, during the meeting sessions, the clinicians showed great enthusiasm and interest in the opportunity to deepen their knowledge of immersive VR and the characteristics of the serious game, asking several questions on the application. This result is notable since neuro and psychomotor therapists are used to practicing their profession in environments where technological instrumentation is not usually privileged.

Specifically, regarding the TAM, the correlations reported in Table 3 showed a strong link between most of the examined variables and the system's perceived usefulness (PU), indicating that the system was meaningful for the therapists. IMG and BI, instead, were not correlated. Regarding BI, a possible explanation may be related to the remote testing of the application or the fact that the choice of using it was rather related to factors not directly detected by the questionnaire (e.g. the engagement level, the ecological setting).

IMG is often a scale related to social and cultural background. It is possible that the therapists could not express the same opinions as the end-users, or that VR is getting quite common for the administration of rehabilitation interventions (although novel, it cannot be defined as a ‘status symbol’, considering its cost-effectiveness).

On the other hand, however, IMG results correlated with OUT, indicating that the quality of the system and the visual and audio feedback were appreciated, also by the therapists. Also, the demonstrability of the results (RES) that could be obtained using the application emerged clearly despite the remote testing. The significant correlations of the subscale RES with PU and PEOU suggested that the application was user-friendly, intuitive and potentially meaningful for children with ADHD. Also, given the correlation of RES with ENJ, we may argue that the engagement provided by VR also played a key role in the positive feedback provided by the therapists.³⁷ Given this, we may affirm that our hypotheses regarding both ADAD ecological validity and its capability of engaging the users were demonstrated, at least in a sample of therapists.

Beside confirmations on the usefulness and potential of the application for the therapists, in accordance with the results of the TAM questionnaire, several insights on how to implement additional features and improvements were also collected in the semi-structured interview. Among these, one of the most mentioned suggestions for improvement was changing the guiding voice with a more natural one. It has also been proposed to use the therapist's voice. The second most common theme concerned giving the therapist the chance to adjust the timing and the task difficulty. This last request emerged because children with ADHD and other attentional disorders can easily lose interest in an activity performed if it is for too long or not challenging enough.^38,39 In this regard, some comments referred to the creation of different scenarios to make the game even more captivating or the insertion of a more explicit goal in the game (e.g. collecting the objects you need to prepare a birthday party).

Another point addressed by participants was about the therapist's role during the game session. Some other comments focused on the type of grip and the gestures to perform with controllers to interact with virtual objects, as these were not visible during the remote test.

Besides the few criticisms and suggestions for improvements, therapists reported that it was clear that the environment had been conceived according to specialists’ indications and that it answered the needs of therapeutic practice. Given this, it seems possible to state that this preliminary study represents a step forward in applying VR technology to rehabilitate children with attention deficits.

According to the literature, the advantages of VR include facilitating the transfer of the acquired abilities into daily life; thus, the increase of ecological validity has important consequences in terms of motivation and adherence to treatment.^6,7,40,41 According to therapists, this objective could be fulfilled with ADAD, thus further studies are required to assess whether this hypothesis is confirmed in a sample of children with ADHD.

The collected outcomes were positive regarding the secondary aim, that is, the possibility of assessing the acceptance and usability of an application using a remote teleconference system. Although not compared to a validation performed in presence all the participants were satisfied with what they could perceive about the application and said that they would have been able to perform all the tasks independently.

Although forced by COVID-19 restrictions, this paves the way for the use of teleconference systems also in the future, given the advantages of reaching therapists all over the world, and accommodating their, often strict, time schedules.

Limitations

We acknowledge that this work presents some limitations that can be summarised as follow. The first one is related to the choice of participants. Primarily, the use of a relatively small sample reduces the generalisability of findings. Moreover, participants were recruited using convenience sampling rather than random sampling. In addition, considering that many participants tested this technology for the first time, a ‘novelty effect’ may have affected the results.⁴² Anyway, the consistency of the data collected as the Cronbach alphas’ values and the existence of meaningful correlations between the examined variables represents positive evidence of reliability. The COVID-19 pandemic also affected the study, forcing us to perform the evaluation remotely through the Skype platform due to the restrictions in accomplishing face-to-face activities. Nonetheless, even during this challenging period, we could reach the target population and gain insights on the feasibility of using such kind of application for rehabilitation purposes.

Future perspectives

Further development of this study will include, according to the improvements that emerged from the validation with experts necessary to develop a fully functional application, a secondary validation phase involving healthy children and, following, children with ADHD. This will enable us to compare results from the two cohorts, highlighting peculiarities and specific acceptance aspects characterising children with ADHD. Moreover, it will be possible to better investigate the feasibility of introducing the VR-based application and related equipment as a supporting tool for clinicians in school-aged children treatment and identify the most appropriate age group to propose the intervention.

Possible improvements to the systems can also be from the technological point of view, particularly concerning the possibility of handling the level selection (namely the difficulty level) according to the child's capabilities. As highlighted in the literature regarding the rehabilitation of ADHD, the treatment should indeed be personalised according to the specific case, based on age, secondary disorders and the severity of symptoms.²⁹ In fact, this disorder is very heterogeneous and with different characteristics depending on the determined child; for this reason, it is essential to start with an accurate and personalised therapeutic intervention for the individual patient.

Conclusions

This work aimed at studying the acceptance and usability of a VR-based application developed to support clinicians in the treatment of children with ADHD. As highlighted by literature, VR represents a valuable means for rehabilitating children with such disorders due to its great ecological validity and the possibility to create engaging and motivating customisable environments according to the patient's specific needs.^37,43

The results of this work pave the way for future validation of ADAD with children: the active participation of therapists in the online sessions and their unanimous favourable judgement allowed us to conclude that the application was well accepted and usable for the experts.

Although the study was performed online due to the pandemic situation, participants confirmed that the actions performed by the operator were clear and made it possible to understand the tasks required within the serious game. This provides evidence for the feasibility of additional expert-based validation from remote. Given the ease of use and the spreading of teleconferencing systems, also boosted by the COVID-19 pandemic, we believe that the same procedure could be effective and applicable to the medical field.⁴⁴ Indeed, even if this work represents a preliminary study, the mediated experience represented a valuable approach to support the design process of VR-based applications enabling experts from specific domains to suggest possible improvements during all the development phase.

Also from a practical and educational point of view, such a method can be useful to provide medical personnel with more information and increase their awareness regarding the potential of new technologies such as VR.