THE LIMITATIONS AND CHALLENGES IN THE ASSESSMENT OF EXECUTIVE DYSFUNCTION ASSOCIATED WITH REAL-WORLD FUNCTIONING: THE OPPORTUNITY OF SERIOUS GAMES

Abstract

Nowadays, there is a broad range of methods for detecting and evaluating executive dysfunction ranging from clinical interview to neuropsychological evaluation. Nevertheless, a critical issue of these assessments is the lack of correspondence of the neuropsychological test’s results with real-world functioning. This paper proposes serious games as a new framework to improve the neuropsychological assessment of real-world functioning. We briefly discuss the contribution and limitations of current methods of evaluation of executive dysfunction (paper-and-pencil tests, naturalistic observation methods, and Information and Communications Technologies) to inform on daily life functioning. Then, we analyze what are the limitations of these methods to predict real-world performance: 1) A lack of appropriate instruments to investigate the complexity of real-world functioning, 2) the vast majority of neuropsychological tests assess well-structured tasks, and 3) measurement of behaviors are based on simplistic data collection and statistical analysis. This work shows how serious games offer an opportunity to develop more efficient tools to detect executive dysfunction in everyday life contexts. Serious games provide meaningful narrative stories and virtual or real environments that immerse the user in natural and social environments with social interactions. In those highly interactive game environments the player needs to adapt his/her behavioral performance to novel and ill-structured tasks which are suited for collecting user interaction evidence. Serious games offer a novel opportunity to develop better tools to improve diagnosis of the executive dysfunction in everyday life contexts. However, more research is still needed to implement serious games in everyday clinical practice.

INTRODUCTION

Nowadays, there are different views of executive functions (EF), but it is common to refer to it as an “umbrella concept” due to the multidimensionality of its nature (Goldstein et al., 2014). EF is one of the most complex cognitive domains of human behavior, processes that underlie goal-directed behaviors (Best & Miller, 2010; Burgess et al., 2006; Chan et al., 2008; Chaytor et al., 2006; Gioia & Isquith, 2004; Manchester et al., 2004). EF is a skill essential for physical and mental health; success in life (school, work, family); and cognitive, social, and psychological development (Diamond, 2020; 2013).

Some authors define EF mainly centered in the cognitive process involved, such as self-monitoring, task-setting, working memory, problem solving, and planning (Burgess, 2004). Although EF as a neuropsychological construct has long been considered a unitary, general purpose higher-order ability, most authors proposed that it is more accurately characterized as a collection of three core processes: inhibition, working memory, and cognitive flexibility (e.g., McAlister et al. 2016; Lehto et al. 2003, Miyake et al. 2000). From these, higher-order EF are built such as problem-solving, reasoning, and planning (Collins & Koechlin 2012, Lunt et al. 2012)

However, other authors have proposed that EF also refers to the ability to monitor and modulate emotion and behavior include in EF both cognitive processes (the cold component of EF) and self-regulation of behavior (the hot component of EF) (Chan et al., 2008; Stuss, 2011).). As stated by Lezak (1995) “EF refer to a collection of interrelated cognitive and behavioral skills that are responsible for purposeful, goal-directed activity, and include the highest level of human functioning, such as intellect, thought, self-control and social interaction”.

In practical terms, EF acts when the person needs to take an active position in controlling their behavior, going from automatic processing to the formulation planning and execution of objectives (Stuss, 2009). EF comprises a wide range of cognitive subdomains and behavioral competencies, such as cognitive flexibility, the ability to move from one task to another (“shifting”), the inhibition of automatic responses not appropriate to the context, the ability to keep in mind the information necessary to carry out the ongoing actions, the ability to modify behavior to adapt to the context, multitasking, problem-solving, resistance to interference, self-awareness, metacognition, and the capacity to address novelty (Burgess, Veitch, de Lacy Costello, & Shallice, 2000; Chan et al., 2008), among others.

This manuscript will consider EF in this wide meaning (cognitive processes and self-regulation) because it is now well established that EF depends on multiple and distributed systems (Duncan, Assem, & Shashidhara, 2020)

Executive dysfunction is defined as impairment of EF. However, there is a trend to use the term “executive dysfunction”, a psychological construct, and “frontal lobe dysfunction”, an anatomical syndrome, as two interchangeably concepts, leading to a conceptual confusion (Stuss, 2011). In this paper, we use the concept “executive dysfunction” as a psychological construct.

In neuropsychiatric disorders, executive dysfunction has been described as the main factor associated with functional impairment, which are manifested as problems with managing daily routine or emotional lability. Executive dysfunction is experienced by persons with brain disorders independently of the cause of their disease (Cieza et al., 2015; Sabariego et al., 2015; Slachevsky, Reyes, Rojas, & Silva, 2009). Moreover, functional impairment due to executive dysfunction is observed in neurological disorders (Royall et al., 2002; Sabariego et al., 2015) such as brain tumor (Robinson et al., 2014), traumatic brain injury (Karr, Areshenkoff, Duggan, & Garcia-Barrera, 2014), cerebrovascular disease (Hua et al., 2014), Parkinson disease (Gasca-Salas et al., 2014), Multiple sclerosis (Phillips et al., 2014), Tourette syndrome (Eddy, Rickards, & Cavanna, 2012), attention-deficit/hyperactivity disorders (Sebastian et al., 2014), and dementia (Marshall et al., 2011; Moheb, Mendez, Kremen, & Teng, 2017), but also in psychiatric disorders, for example schizophrenia (Xu et al., 2014), or obsessive-compulsive disorder (Nakao, Okada, & Kanba, 2014), among others. Moreover, a shared set of functional impairments and environmental factors experienced across different brain disorders has been proposed as the hypothesis of the so-called medical “horizontal epidemiology” by the European PARADISE consortium (Cieza et al., 2015). This hypothesis suggests that functional impairment and environmental factors are shared by nine brain disorders (epilepsy, migraine, multiple sclerosis, Parkinson disease, stroke, dementia, schizophrenia, substance dependence), which are assessed by the use of an extensive protocol in the different European populations (Cieza et al., 2015).

Therefore, regardless of the pathology that produces it, executive dysfunction might cause people to be significantly affected in their ability to manage themselves in their daily context effectively. Thus, an adequate assessment of executive dysfunction might be a good predictor and indicator of the functional impairment (Adamit et al., 2015; Garrett et al., 2019; Romero-Ayuso et al., 2019), regardless of age (Chan et al., 2008; Hosenbocus & Chahal, 2012) or the severity of the underlying disease (McDonald, Flashman, & Saykin, 2002). Even though there is a substantial number of EF assessment methods, there is consensus that there are numerous limitations and barriers to effectively predicting functional impact (Paul W. Burgess, 1997; Paul W Burgess et al., 2006b). As we will show in the manuscript, three main challenges remain in the assessment of the real-world functioning: i) there is a lack of appropriate instruments to investigate the complexity of real-world functioning (Schmitter-Edgecombe & Arrotta (2022), because i) EF are usually assessed as an isolated cognitive domain disconnected from the social and intersubjective context (content of evaluation). ii) most EF tests assess tasks with clearly specified criteria. This contrasts with real-world situations, which typically lack well-defined criteria for evaluating whether the goal has been met (task structure). iii) measurements of behaviors are based on simplistic data collection and statistical analysis (measurement of performance).

To address the challenges in the assessment of real-world performance, this paper proposes serious games as a new framework to improve the detection of real-world functioning caused by executive dysfunction. Thus, we show three topics consecutively in the manuscript: First, we will briefly review the current methods of evaluation of executive dysfunction. Second, we will describe the main limitations of these methods to predict real-world performance (content of evaluation, task structure, and measurement of performance). Finally, we will show how the use of Information and Communication Technologies (ICT), and more specifically, the use of serious games based on screen-based simulations, offer a novel opportunity to develop better tools to improve diagnosis of executive dysfunction in everyday life contexts. Serious games provide meaningful narrative stories and virtual or real environments that involve the user in goal-directed actions in natural and social environments with social interactions (Przybylski et al., 2010). Serious games allow the creation of interactive social contexts more naturalistic and close to every life situation, generating an atmosphere as if the person is in the intersubjective social world (Poulin et al., 2013).

CURRENT ASSESSMENT OF EXECUTIVE FUNCTION

Nowadays, there are different proposals for evaluating EF associated with real-world functioning (Chan et al., 2008; Verdejo-García & Bechara, 2010), making it clear that there is no single instrument for this purpose. The assessment methods of executive dysfunction are the clinical interview and the traditional neuropsychological assessment. Currently, these two methods are the most informative to determine cognitive and behavioral sequelae of brain diseases (Bigler, 2016), given that brain imaging studies, by themselves, show a minimum value when predicting such deficits in the clinical context at the individual level (Slachevsky et al., 2009; Slachevsky, Ramos & Olavarría, 2022). For this article, we are going to focus on neuropsychological assessment only (Table 1 summarizes some of the main evaluations of executive dysfunction).

Table 1:

Tests to detect executive dysfunction

Paper-and-pencil tests	Naturalistic observation	ICT
Test based on specific cognitive domains: Rey Complex Figure Test (Osterrieth, 1944) Rivermead Behavioral Memory Test (Wilson et al., 1985) Test of Everyday Attention (Robertson et al., 2001) Wisconsin Card Sorting Test (WCST) (1996) Trail Making Test (TMT) (1958) Neuropsychological batteries: Wechsler Adult Intelligence Scale (WAIS) (1939) Luria-Nebraska Battery (Golden et al., 1985) Questionnaires: Behavioral Assessment of the Dysexecutive Syndrome (Wilson et al., 1996). Behavioral Dysexecutive Syndrome Inventory (BDSI) (2010) The Neuropsychiatric Inventory (NPI) (1994)	Multiple Errands Test -MET- (Shallice and Burgess, 1991) Executive Function Route-finding Task -EFRT- (Boyd and Sautter, 1993) Executive Function Performance Test -EFPT- (Baum et al., 2008; Weiner et al., 2012) Functional Independence Measure -FIM- (Keith, 1987).	Computer-based neuropsychological assessment: Tower of Hanoi (Mataix-Cols & Bartrés-Faz, 2002) the Category Test (Choca & Morris, 1992) Raven Progressive Matrices Test (Williams & McCord, 2006) CANTAB (Sahakian & Owen, 1992), Virtual Reality V-Store (Lo Priore et al., 2003) Virtual Action Planning-Supermarket -VAP-S- (Klinger et al., 2004) Virtual Mall (Rand et al., 2005) Virtual MET (Rand et al., 2009) Serious Games Virtual Reality Grocery (V-Store) (Levy et al., 2019) The multitasking in the city test -MCT- (Jovanovski et al., 2012a, 2012b) Non-immersive Virtual Coffee task (NI-VCT) Besnard et al., 2016).

Over time, the neuropsychological assessment of executive dysfunction has evolved into three different main methods.

1. The first method, and probably the most used, is the paper-and-pencil tests, and there are numerous specially designed for the evaluation of EF. While some of them are intended to measure a single subdomain of EF, other ones seek to measure several aspects of this cognitive area (Burgess, 2004a). Although clinicians widely use them, paper-and-pencil tests are not exempt from limitations. Firstly, this assessment requires the physical presence of an expert evaluator (Brearly et al., 2017; Valladares-Rodríguez, Pérez-Rodríguez, Anido-Rifón, & Fernández-Iglesias, 2016), restricting access to neuropsychological services depending on the country’s health system (Adjorlolo, 2016; Jacobsen, Terje, Stein, & Jan-Magne, 2003). Another critical issue is that the ecological validity of these tests to detect real-world performance could be improved (Chaytor & Schmitter-Edgecombe, 2003; Marcotte & Grant, 2009; Sbordone, 1996; Sbordone et al., 1998; Sharma et al., 2014). For instance, it was found that performance of EF tests (i.e., Wisconsin Card Sorting Test, Trail Making Test, Stroop, and Controlled Oral Word Association Test) accounted for 18% to 37% of the variance in daily life activities (McAlistair et al 2016; Schmitter-Edgecombe & Burr, 2006). Despite these associations, a large proportion of variance in functional outcomes is still not explained by performances in these cognitive tasks showing that they present a limited capacity to predict daily functioning (McAlister et al., 2016). For instance, some people with significant cognitive impairment maintain a good level of real-world functioning, while others with less cognitive impairment are more functionally disabled. Thus, there is clear that the ability to function in onés environment is determined by non-cognitive factors (e.g., social, physical, environmental), which contribute to the maintenance of real-world functioning (Farias et al., 2020). It is important to highlight that real-world functioning requires other than cognitive and emotional functions, physical or sensory functions, among others, are needed for everyday life functioning and achieve a comprehensive evaluation in real-life contexts.

   To overcome this limitation in the paper-and-pencil tests, it has been proposed to apply instruments with ecological validity, which means that the neuropsychological test can predict the behavioral outcomes in real life (Franzen & Wilhelm, 1996). In recent decades, neuropsychology has advanced with two fundamental strategies in developing ecologically valid evaluation: veridicality and verisimilitude (Franzen & Wilhelm, 1996). The veridicality approach suggests that the EF tests, which were not designed from an ecological perspective, are valid measures to predict the functional capacity of the subject in their daily lives (Franzen & Wilhelm, 1998). To do this, they study the correspondence established between these tests and tools that assess functional aspects of daily life. The verisimilitude approach (also referred to as “representativeness” (kvavilashvili & Ellis, 2004) claims that the degree of cognitive demand of a test must reproduce the cognitive demand that the subject performs in the activities that he develops in his day to day (Chaytor & Schmitter-Edgecombe, 2003). Although it is generally assumed that the tests based on the verisimilitude approach have higher face validity than those tests based on a veridicality approach (Zartman et al., 2013), recent research has demonstrated that they do not necessarily have greater ability to predict functional independence (Ziemnik & Suchy, 2019).

2. Subsequently, instruments were developed based on naturalistic observation, which consists of an observational assessment while the patient performs specific daily life activities. Some of these are the Multiple Errands Test -MET- (Shallice & Burgess, 1991), the Executive Function Route-finding Task -EFRT- (Boyd & Sautter, 1993), the Executive Function Performance Test -EFPT- (Baum et al., 2008; Weiner, Toglia, & Berg, 2012), and the Functional Independence Measure -FIM- (Keitll, Granger, & Hamilton, 1987). This assessment method is generally developed in facilities specially designed to simulate real environments (Giovannetti, Libon, & Hart, 2002) and focused on measuring the participant’s activity and participation level rather than their impairment level (Poulin, Korner-Bitensky, & Dawson, 2013). These real-life assessments have shown to be more accurate than assessments developed in laboratory settings (Burgess et al., 2006; Rand et al., 2009). Nevertheless, they present some limitations, such as that they are time-consuming and it is challenging to obtain standardized measures among administration centers (Chevignard et al., 2000; Fortin, Godbout, & Braun, 2003).

3. Finally, with the explosive growth of Information and Communication Technologies (ICT), evaluations based on ICT have emerged as a new method to overcome the inherent limits of traditional neuropsychological assessment (Burgess et al., 2006). The use of ICTs has become an innovative methodology, which has been termed the “Neuropsychological Evaluation 3.0” (Parsons, 2016). Nowadays, there are three primary assessment methods based on ICTs: Computer-based neuropsychological assessment; virtual reality; and serious games.

   1. Computer-based neuropsychological assessment: This is an assessment procedure similar to the paper-and-pencil tests but administered by computational support. This method focuses on identifying impairments in complex cognitive and behavioral abilities, such as memory, EF, and activities of daily living (Edwards, Vess, Reger, & Cernich, 2014). Some examples of computerized versions applied to assess EF are the Tower of Hanoi (Mataix-Cols & Bartrés-Faz, 2002), the Category Test (Choca & Morris, 1992), the Raven Progressive Matrices Test (Williams & McCord, 2006), and the CANTAB (Sahakian & Owen, 1992), which have shown a high correlation in administration times and scores obtained compared to paper and pencil tests (Choca & Morris, 1992; Hoskins, Binder, Chaytor, Williamson, & Drane, 2010; Mataix-Cols & Bartrés-Faz, 2002; Sahakian & Owen, 1992; Schatz & Browndyke, 2002; Williams & McCord, 2006). Since most computerized neuropsychological tests just automate construct-driven paper-and-pencil assessments, they show similar limitations to predict a person’s real-world functioning (Parsons, 2016).

   2. Virtual Reality (VR): This is a technological assessment that simulates real-life situations with the potential to have higher ecological validity (Parsey & Schmitter-Edgecombe, 2013; Parsons, 2011). This evaluation allows the user to immerse themselves in interactive three-dimensional environments that reproduce natural environments and situations that directly measure the functional limitations caused by neuropsychological impairments. For example, some of the tasks that have been used to detect executive dysfunction associated with a functional impairment through VR are: to perform an errand task in a three-floor building (McGeorge et al., 2001), to buy fruit in a store(V-Store) (Lo Priore, Castelnuovo, Liccione, & Liccione, 2003), to make a purchase in a supermarket (VAP-S, VMall, VMET) (Klinger, Chemin, Lebreton, & Marié, 2004; D. Rand, Katz, Shahar, Kizoni, & Weiss, 2005; Debbie Rand et al., 2009), to prepare soup in a kitchen setting (Zhang et al., 2001), and the development of different tasks both in the house and the city (Kourtesis, Korre, Collina, Doumas, & MacPherson, 2020), among others. The performance on VR is significantly associated with computerized and pencil-paper assessments (e.g., Armstrong et al., 2013). Limitations of VR to predict real-world functioning will be reviewed in “Future perspectives: the opportunity of serious games”.

   3. Serious games (SG): A third alternative for assessing executive dysfunction through the ICTs is the use of serious games. SG are defined as specialized digital games whose main purpose is not entertainment (Michael & Chen, 2005; Robert et al., 2014). At present, the use of SG for purposes other than mere enjoyment has been well received in the field of rehabilitation (Legouverneur, Pino, Boulay, & Rigaud, 2011; Manera et al., 2015; 2017; Martínez-Pernía, González-Castán, & Huepe, 2017), as a virtual training simulator to improve real-life skills (Vallejo et al., 2017), and as a therapeutic tool to improve behavioral disturbances associated with the EF (e.g., apathy (Robert, Manera, Derreumaux, Montesino, Leone, Fabre & Bourgeois, 2020; Robert, Albrengues, Fabre, Derreumaux, Pancrazi, Luporsi & Manera, 2021)). Besides, several SG developments have been applied in children, adults, and older adults (healthy and unhealthy populations) to assess executive dysfunction (e.g., Wang et al., 2022; Valladares-Rodriguez et al. 2019; Neto et al., 2018; Weijer-Bergsma et al., 2015; Jansari et al., 2014). For instance, Levy and colleagues (Levy et al., 2019) created a virtual environment with a seven-aisle grocery story (V-mart) where participants are required to perform different tasks (time management, budgeting, planning) of increasing complexity. Another example is the multitasking in the city test -MCT- (Jovanovski, Zakzanis, Campbell, Erb, & Nussbaum, 2012; Jovanovski, Zakzanis, Ruttan, et al., 2012) where participants have to perform specific errands while moving through the city composed of different services, shops, and the participant’s home. Another example is the Non-immersive Virtual Coffee task (NI-VCT), where participants have to make a cup of coffee with a coffee machine by selecting from a virtual workbench the items necessary to achieve the task (Besnard et al., 2016). Finally, the whack-a-mole game consist of hitting any moles that appeared in the screen (Tong et al., 2016; Tong et al., 2015; Tong et al., 2014), avoiding other targets (moles with hats and butterflies) (Tong, Chignell & DeGuzman, 2021). Furthermore, different studies developed with SG have shown both good psychometric properties and ecological validity, measured with naturalistic event-based tasks and in the real-word, in the assessment of the executive dysfunction, such as the Video Assessment of Prospective Memory -VAPM- (Clune-Ryberg et al., 2011), the Virtual Interactive Shopper -VIS (Hadad et al., 2012), the Virtual Library Task -VLT- (Renison, Ponsford, Testa, Richardson, & Brownfield, 2012), the Virtual Reality automated Teller Machine -VR-ATM- (Fong et al., 2010), and the Virtual Reality-Based Prospective Memory Training Programme -VRPM- Assessment Scenario (Yip & Man, 2013). Limitations of SG to predict real-world functioning will be reviewed in “Future perspectives: the opportunity of serious games”.

Despite the advances during last years, where the neuropsychological assessments (traditional and technological) have shown a good external validity to predict real-world functioning, they still have substantial limitations reflecting a relatively low capacity to predict the impact of executive dysfunction in daily life activities (Martínez-Pernía et al., 2017; McAlister, et al, 2016; Chaytor & Schmitter-Edgecombe, 2003; Chaytor et al., 2006; McGeorge et al., 2001; Sbordone, 2010; Spooner & Pachana, 2006; Tupper & Cicerone, 1990; Zhang et al., 2001), as it is to face the challenges of the workplace, the academic field or interpersonal relationships, whose rules of action are not explicit and have an important demand for EF.

LIMITATIONS OF CURRENT NEUROPSYCHOLOGICAL INSTRUMENTS FOR THE EVALUATION OF EXECUTIVE DYSFUNCTION IN THE DETECTION AND PREDICTION OF FUNCTIONAL IMPAIRMENT

Despite recent progress in the development and refinement of neuropsychological instruments to evaluate executive dysfunction, considerable challenges remain. We summarize three key issues below.

i). Content of evaluation

While there has been a long history of assessments for investigating specific aspects of EF, such as cognitive flexibility or set-shifting ability (e.g., (Grant & Berg, 1948; Reitan, 1958), other aspects of EF have traditionally received far less attention. This includes domains such as prospective memory, which requires both memory for the content of delayed intentions and also EF due to the need to inhibit an ongoing activity and switch to the intended action at the appropriate time (Kourtesis & MacPherson, 2021; Fish, Wilson, & Manly, 2010). Other examples of relatively-neglected areas include contextual control (Ibañez & Manes, 2012), and multitasking (P. W. Burgess, 2010). Each of these could be associated with functional impairment associated with executive dysfunction; however the relative paucity of appropriate measures for assessment makes this hard to evaluate. It should also be noted that executive dysfunction is also likely to play a role in domains that are not traditionally considered to fall within the definition of EF, for example, aspects of social cognition (Adolphs, 1999; Amodio & Frith, 2006), emotion regulation (Goldin, McRae, Ramel, & Gross, 2008), and creativity (Gonen-Yaacovi et al., 2013). A lack of appropriate instruments to investigate these aspects of cognition can also lead to a failure to evaluate functional impairment appropriately. Furthermore, even when these domains have been evaluated, they are typically considered in isolation. This contrasts with real-world situations in which cognitive demands across multiple domains must be integrated (e.g., tasks that involve both social and executive demands). As a result, the standard tools for evaluating executive dysfunction could mask difficulties that only become apparent when multiple demands, across distinct cognitive domains, need to be integrated (M.-M. Mesulam et al., 2014). In sum, while typical neuropsychological tests are valuable as a way of identifying selective cognitive deficits, this approach can be complemented by tests that assess additional socio-cognitive domains which are not typically assessed, as well as the manner in which these domains are integrated within standardized assessment protocols.

ii). Task structure

The vast majority of tests used for neuropsychological assessment are “well-structured” (for discussion of this concept, see (Gilbert, Zamenopoulos, Alexiou, & Johnson, 2010; Goel & Grafman, 2000). That is, the goal of the task, the set of appropriate responses, and the criteria to evaluate whether or not the goal has been achieved are clearly specified. This contrasts with “ill-structured” tasks, which typically lack well-defined criteria for evaluating whether the goal has been met. Ill-structured tasks may also be open-ended in the sense that it is not obvious at what point the task has been completed. Although the ultimate goal may be imposed by an experimenter, it is up to the individual performing the task to set their own criteria for whether or not they have been successful, and when the task is complete. Although ill-structured tasks are rare in neuropsychological assessment, this type of task may be much more characteristic of everyday life than the standard assessment procedures used by neuropsychologists. Patients with frontal lesions may perform relatively poorly on ill-structured tasks such as those involving creative planning (Goel & Grafman, 2000) and multitasking (Burgess, 2000; Shallice & Burgess, 1991), even in the context of good performance on other well-structured tests of EF. Therefore, testing on well-structured tasks alone may miss the difficulty that some patients have with ill-structured ones, as found by Shallice and Burgess (1991) in their Multiple Errands Test, an ill-structured planning task. One characteristic of the ill-structured tasks is that the neuropsychological assessment allows different courses of action that participants might take, and they must decide for themselves what approach they will take. These tasks stress the critical executive ability to find for themselves a way of approaching that makes it easier or makes their performance more effective (Burgess & Shallice, 1996). Different studies have shown that patients with executive dysfunction performed less effective strategies than healthy groups (White, Burgess & Hill, 2009; Burgess & Shallice, 1996; Alderman, Burgess, Knight, & Henman, 2003).

iii). Measurement of performance

The arguments above highlight the utility of developing new assessment methods that can provide rich multidimensional datasets in the context of ill-structured tasks. Such methods will also require new analysis techniques. This is for two key reasons. First, whereas traditional tasks such as the Trail Making Test yield only a small number of measures (e.g. completion time) which can be analyzed separately with a univariate approach, more complex datasets with a large number of measures will require multivariate methods (see e.g. Kourtesis & MacPherson, in press, for an example from the domain of prospective memory). A second issue when it comes to ill-structured tasks is that, by definition, such tasks lack clear criteria according to which an individual’s performance can be scored as “correct” or “incorrect”. Instead, it is up to the individual to set their own criteria. In order to link functional outcomes or diagnostic categories with multidimensional datasets from ill-structured tasks, novel statistical approaches will be required. One promising approach is to use machine learning methods to reduce the dimensionality of complex datasets (Alonso-Fernandez, 2019; Karapapas & Goumopoulos, 2021).

Therefore, there are still critical deficiencies in detecting and predicting functional impairment associated with executive dysfunction. An analysis resulting from the insights in the previous paragraphs is related to the fact that these assessment methods consider the EF in a narrow sense. This means, on the one hand, that these methods focus on assessing some specific subdomains of the EF, while other subdomains are rarely considered part of real-world functioning (e.g., contextual control, decision-making intersubjective contexts, multitasking). On the other hand, and more critically, these evaluation methods prioritize the understanding of EF as cognitive processes depending on mechanistic and logical tasks, but they omit those cognitive processes influenced by social and interactional context. It is worth noting that, for example, the diagnostic criteria for “Major neurocognitive disorder” (DSM V) require clinicians to demonstrate a substantial impairment in cognitive performance, documented through neuropsychological testing, and that such impairments interfere with independence in daily functioning, requiring assistance with activities of daily living (American Psychiatric Association, 2013). Hence, such criteria expect an association between the type and severity of cognitive decline and the loss of functional abilities. Unfortunately, such an association is rarely found relying on available assessment forms. Nevertheless, we are witnessing the emergence of functional assessments, which capture the potential contribution of domain-specific cognitive decline to impairments in everyday life functions (e.g., Farias et al., 2013). The search of more ecologically valid cognitive tests and theory-driven functional scales would allow the assessment of functional abilities distilling the contributions of the underpinning cognitive processes and the influences of social and interactional contexts.

Predicting real-world behavior based only on non-social tasks is a key deficiency that needs to be addressed in neuropsychological assessment; because the ability to interpret and coordinate behaviors in social situations is one of the main functions of prefrontal cortex and EF (Hari & Kujala, 2009), and this ability enables appropriate behavior in daily life (Ibañez & Manes, 2012). Nowadays, the neuropsychological evaluation assesses the performance of the person embedded in the environment (e.g., buying food in the supermarket, cooking in a kitchen, self-caring in the house). However, it does not assess how the social-interaction influences EF in everyday life (e.g., regulation of one’s social behavior, decision-making involving emotional interpretation, multitasking in unstructured environments, resistance to social-context-interference, understanding explicit and implicit contextual keys, among others). Mesulam (1986) argues that conventional neuropsychological tests are insensitive to executive dysfunction because their diagnostic capacity places cognition as a phenomenon outside the person’s daily activity and of social context. Therefore, traditional tests are limited for the evaluation of real-world dysexecutive function because their cognitive demands do not integrate fundamental cognitive and emotional processes into the carrying out actions in everyday life (Paul W. Burgess, Alderman, Volle, Benoit, & Gilbert, 2009), limiting the ability to infer the meaning of the performance of these tests to the personal and social conditions of the evaluated individual. In addition, Damasio (1994) points out that these evaluation methods lack ecological utility as “instruments of rationality”, which means that the mind is understood in logical and abstract terms (words, numbers, objects, space, temporality). Some authors consider that the neuropsychological tests do not evaluate cognition according to the facilities or difficulties that a person presents in their daily lives but only as a disembodied cognition, separated from the social and interpersonal context (Damasio, 1994; Martínez-Pernía, 2020).

Therefore, and to improve the detection and prediction of functional impairment, we believe that a neuropsychological assessment that considers EF in a broad sense is necessary. That is an assessment that, in addition to assessing simultaneously the different components of EF (Chan et al., 2008; Schwartz, Reed, Montgomery, Plamer, & Mayer, 1991; Shallice & Burgess, 1991; Barbara Wilson, Alderman, Burgess, Emslie, & Evans, 1996), assesses how these processes are influenced by the social context and social interaction. In the following section, we will show how information and communication technology (ICT), and more specifically, SG, offers an opportunity to develop tools to more efficiently diagnose executive dysfunction in everyday life contexts.

FUTURE PERSPECTIVES: THE OPPORTUNITY OF SERIOUS GAMES

In the scientific community, there is a consensus to develop tools for the more efficient evaluation of executive dysfunction in real-life contexts and with a greater capacity to predict real-world functioning (Paul W. Burgess, Alderman, Evans, Emslie, & Wilson, 1998; Paul W Burgess et al., 2006b). Given the recent analysis presented in the previous section, we consider that the two most promising lines of research related to ICTs are: on the one hand, immersive technologies (VR and augmented reality) and on the other hand, SG. However, the frontier between both methods is not always straightforward (e.g., Corti et al., 2021), and besides, there are SG that use immersive technologies (e.g., (Sánchez-Herrera-Baeza et al., 2020; Vogiatzaki & Krukowski, 2014).

The main characteristic of immersive technologies is that they immerse the user in a 3D virtual world, either entirely in the VR or combined with the real world in augmented reality (Cipresso, Giglioli, Raya, & Riva, 2018). Immersive technologies, and especially VR, are being used in the detection of executive dysfunction, but above all, they are being applied extensively in the creation of new treatments and interventions in neurological disorders (Ghai, Ghai, & Lamontagne, 2020; Saredakis et al., 2020). For instance, VR memory training improved cognitive abilities in participants navigating along with virtual environments with a head-mounted system (Optale et al., 2010). However, despite immersive technologies’ evolution, they are still not completely mature and have not yet been widely adopted in society, making their application more costly and difficult in neuropsychological assessment (A. Rizzo & Koenig, 2017). Moreover, VR still has critical issues related to body-machine interactions, social interactions, and technological developments that hinder its simple implementation in detecting real-world behavior. Below, we explain these three limitations.

1. Concerning body-machine interactions with VR, it has been shown that the performance of a test in a semi-closed environment (VR glasses) could produce anxiety, preventing participants (cognitively impaired and healthy) from completing the assessment (M. Rizzo, Sheffield, Stierman, & Dawson, 2003; Saredakis et al., 2020; Stanney, Kingdon, & Kennedy, 2002). Besides, the use of head-mounted display systems causes motion sickness (dizziness, nausea, headache, instability, fatigue), which endangers the health and safety of the participants (Parsons, McMahan, & Kane, 2018), affects behavioral and cognitive performances (Nalivaiko, Davis, Blackmore, Vakulin, & Nesbitt, 2015), and decreases the reliability of data (Kourtesis, Collina, Doumas, & MacPherson, 2019). Additionally, most VR produces a strong sense of stimulation and immersion to generate a vivid virtual environment, which may be unsuitable in some populations (e.g., people with heart diseases) or, being cautious in its use (e.g., elderly people) (Ning, Li, Ye, Zhang, & Liu, 2020).

2. Concerning social interactions, realistic social interactions are not yet possible to implement in VR (Pan & Hamilton, 2018). This is caused because i) the avatars’ appearance and behavior are not “truly life-like”, and ii) although game users can interpret avatars’ thoughts and emotions, participants never interact with them (Hermans et al., 2019). These limitations can result in the uncanny valley effect (Kourtesis et al 2020). Concerning this effect, Mori and colleagues (2012, p. 98) stated, “in climbing toward the goal of making robots appear like a human, our affinity for them increases until we come to a valley” (Mori et al., 2012). Therefore, the uncanny valley effect negatively impacts the detection of real-world functioning using VR.

3. Concerning the technological limitations of immersive systems, different factors complicate its clinical implementation: i) VR is a high-cost technology that still has high technical maintenance requirements (Parsey & Schmitter-Edgecombe, 2013), both in the devices needed for their deployment and in the development of the contents. In addition, the lack of widely accepted industry standards means that developments and devices have a shorter life cycle, making VR even more expensive. ii) VR development is still a fragile and complex process that requires high technical expertise, hampering the incorporation of clinical professionals (neurologists and neuropsychologists) (Martínez-Pernía, Núñez-Huasaf, et al., 2017). iii) Immersive systems have specific requirements that limit their application in the clinical field (i.e., lighting and large-scale game devices) (A. A. Rizzo, Schultheis, Kerns, & Mateer, 2004; Werner, Rabinowitz, Klinger, Korczyn, & Josman, 2009).

These issues are possible reasons why VR is still limited to an evaluation method in the experimental phase (Camara Lopez, Deliens, & Cleeremans, 2016), where only a handful of investigations have established virtual environments’ ecological and constructive validity (Besnard et al., 2016). Some authors have stated that the VR ecological validity could not be better than traditional evaluation methods (Chan et al., 2008). As Rizzo and Koening point out for VR technologies, “the majority of conducted studies are pilot trials without sufficient power or the study design needed to draw decisive conclusions about efficacy, transfer of gained skills to the daily life of clients, long-term outcomes, and cost-effectiveness.” (A. Rizzo & Koenig, 2017, p. 888). The literature confirms these aspects because most applications of immersive technologies are mainly exploratory experiments to verify the feasibility, with a limited number of subjects and conducted in controlled environments where there is extensive technical support (Kim, Pang, & Kim, 2019; Park, Kim, Lee, Na, & Jeon, 2019).

As an alternative to the use of immersive technologies to detect real-world functioning, we support the idea that the SG, and more specifically, the screen-based simulations, are potentially the most promising technology for neuropsychological assessment. SG are rapidly becoming an essential tool in the medical domain used with different goals such as improving health behaviors, training, and research (Wattanasoontorn, Boada, García, & Sbert, 2013). These real simulations are developed through a multimodal interaction (de Freitas & Liarokapis, 2011) using animations, texts, language, graphics, haptics, audios, and so on. (Arnab, Petridis, Dunwell, & de Freitas, 2011; Chicchi, Ripoll, Parra, & Alcañiz Raya, 2018; Laamarti, Eid, & El Saddik, 2014; Orozco, Silva, El Saddik, & Petriu, 2012). For instance, they can better include recorded voices and sounds to simulate social interactions in a real-world environment or to include distractors. Other forms of multimodality are possible, such as haptic interaction, but this significantly increases the cost and is more challenging to deploy. Within this realistic simulation, the player has to explore, make decisions, solve tasks, decide among alternative strategies, plan, sequence, interact socially, interpret the behaviors of others, and so on, instead of just physical challenges (Adams & Rollings, 2010) (Figure 1 shows an example of a SG based on screen-based simulations for the evaluation of executive dysfunction). SG are currently employed to support and improve the assessment of different functional and cognitive abilities and provide alternative solutions for patients’ treatment, stimulation, and rehabilitation (Manera et al., 2015). SG have already demonstrated that they can overcome some of the limitations identified with immersive systems and non-technological VR and traditional pen-and-paper-based neuropsychological tests. Below, we will show some advantages of the SG for the detection of executive dysfunction with higher ecological validity. We will show how the use of SG allows overcoming some of the challenges of current neuropsychological assessment and consider the assessment of EF in a broad sense.

Figure 1:

In this serious game, the user has the objective of buying food in the supermarket to cook a recipe. To do the task successfully, the user will need to select products to cook with, interact with vendors to buy the right product, tailor the purchase to available products, interact with other shoppers in uncomfortable or friendly social situations, adjust behavior based on social circumstances and social-emotional interpretation. In more concrete terms, the social interaction in the serious game could occur when the user buys products in the butcher while is being attended by the clerk. In those moments, another shopper (NPC) can verbally advise him to buy a cheaper product by pointing his finger at the item he refers to (explicit and implicit contextual keys). Likewise, shoppers (NPCs) can also be distracted from the task the user is performing by starting conversations or performing physical actions that interfere with goal-directed actions (resistance to cognitive interference in the social context).

As mentioned, the boundaries between SG and VR are blurred, but the most mentioned difference is that SG usually have a more elaborate supporting narrative to enhance players’ relatedness and engagement (Przybylski et al., 2010). As the environments are games or game-like simulations, the gamified aspects are naturally embedded in the game mechanics and interactions and in correlation with the game-supporting narrative. For instance, the game proposes a challenge consistent with the supporting narrative that the user has to achieve by figuring out how it can be achieved and discovering how to achieve it while the game keeps the user informed of the player’s progress or failures. All these characteristics can be used to create evaluation situations in social contexts that are more naturalistic and close to the daily living situations as they can also include social interaction with one or many game characters simultaneously. A SG can simulate the social interaction aspect with the other game characters even though it is a single-player game. This is done by programming one or several predefined behaviors for the other characters that appear in the game (so-called non-playable characters (NPCs)). Thus, when the player approaches NPCs, a set of interactions can be initiated by either the NPC or the player (e.g., a shopper asks for the time to be served at the fruit store). This also allows for the inclusion of social distractors produced by NPCs (e.g., an NPC tries to skip the line to be served first by the fruit platter). As the game has complete control, it is easier to include different stimuli related to social cognition, empathy, emotional regulation, motivation, morality, and emotional recognition in a natural way and incorporate meaningful tasks for the user. Therefore, SG allows for the inclusion of dimensions, all of which directly affect the modulation and processing of the EF, and to assess at the same time other processes involves in the goal-directed actions (e.g., Sacco, Ben-Sadoun, Bourgeois, Fabre, Manera & Robert, 2019). The screen-based simulations allow for accurate reproduction of tasks and procedures in real environments (Martínez-Pernía, Núñez-Huasaf, et al., 2017), generating an atmosphere as if the person is in the intersubjective social world (Lamberts, Evans, & Spikman, 2010; Poulin et al., 2013), and not just embedded in an environment (in a kitchen, in a city).

SG allow the generation of evaluation that include different stimuli in ill-structured tasks that are not always presented in the same form or sequence but that are presented in a way that is still meaningful for the user and overcome limitations of the neuropsychological assessment methodologies based on “well-structured” tasks. Exploration is a natural behavior in games. SG give the opportunity to generate different game dynamics in which the user requires continuous adaptation and innovation in the way he/she responds to the stimuli. This element is fundamental to adequately evaluate the EF as a process that is evaluated in novel situations that the user has not faced before (Chan et al., 2008). SG are capable of generating evaluation processes where simultaneously a narrative of social interaction is generated and where the subject must act through this social influence and his EF skills.

The development of EF assessments relying on SG needs to be rooted in psychometric theories (Bowden, 2017). To such an aim, it is critical to select tasks with adequate construct, content, and ecological validity. Regarding construct validity, we propose assessments that comprise EF tasks which correlate with performance on tasks carried out in environment equivalent to those simulated by SG. Regarding content validity, assessments should involve tasks that: i) index several EF functions such as updating, shifting, and inhibition; ii) tap into cognitive and also emotional processes (e.g., recognition of emotional valence in faces or voices); iii) are ill- structured; and iv) do not require expert knowledge. Finally, v) such tasks should yield multiple metrics to assess performance. By combining the above construct and content validity properties, ecological validity would be likely achieved.

In SG, it is common to track user interaction in detail, obtaining evidence of the actual user behavior, and even adapt the game’s behavior to those interactions. These circumstances make games particularly suitable for neuropsychological assessment. SG can be easily administered and can feature-rich interactive environments to evaluate complex neuropsychological constructs that are difficult to evaluate through traditional tests (Valladares-Rodríguez et al., 2016). It is possible to use game analytics techniques, all the rich in-game user interaction data to estimate the parameters of the underlying cognitive processes, and use the parameters’ values to estimate the user performance (Hagler, Jimison, & Pavel, 2014). SG can include advanced game analytics systems that allow the discovery of complex multidimensional patterns. Game analytics will allow a richer analysis of the data (including machine learning) than some ICT evaluation systems that focus on simpler data such as speed of response, the accuracy of response, or the number of errors (Alonso-Fernández, Calvo-Morata, Freire, Martínez-Ortiz, & Fernández-Manjón, 2019; Valladares-Rodriguez et al., 2018).

SG is a robust and proven technology since the video game industry is the most important in the entertainment field and is present in most homes in the world. There are many game development platforms ranging from professional environments (e.g., Unity3D, Unreal) to authoring environments that simplify the creation of SG (e.g., eAdventure). SG environments such as eAdventure simplify the creation of the games by clinicians with a minimal background in computer science (Martínez-Pernía, Núñez-Huasaf, et al., 2017; Perez-Colado, Perez-Colado, Martínez-Ortiz, Freire-Moran, & Fernández-Manjón, 2017). Also, those environments allow for the deployment of SG on different devices without requiring any additional development (or with minimal ones). For instance, the game can be used on a computer or a mobile device (e.g., tablet or mobile), simplifying the administration at the point of care or in other environments (Valladares-Rodriguez et al., 2019; Tong et al., 2015).

Despite the advantages recently mentioned (for a summary see figure 2), SG do have their drawbacks. For instance, the participant’s interaction with the non-immersive virtual environment does not directly correspond to how their sensorimotor system works in real life, requiring the participant to undergo a learning process to integrate their feeling of ‘being there’ (Corti et al., 2021; Slater, 2009). Another example is that in contrast to the traditional view of assessment methods, where the focus is on assessing specific EF subdomains, our proposal is focused on globally understanding the participant’s real-world functioning in their social and intersubjective environment, losing the possibility of assessing specific EF subdomains. For this reason, we suggest that the best practice to detect executive dysfunction associated with real-world functioning is to complement both assessment methods. Finally, SG are played on electronic devices in front of screens, potentially damaging the useŕs sight. Furthermore, it implies that some populations with poor eyesight could find it hard to interact on the screen (elderly people) (Ning et al., 2020).

Figure 2:

SG offers the opportunity to improve the detection of executive dysfunction in real-world contexts through real simulations based on multimodal interactions, highly interactive environments (natural, social, and interpersonal), ill-structured tasks, advanced game analytics systems, and multiple game development environments for professionals and beginners

However, by the arguments previously indicated, we consider that SG are a mature technology, with potentially high power as a predicting tool of executive dysfunction associated with real-world functioning. However, more research is still needed to implement SG that are reliable, validated in different environments, real-world simulations, and interactional social contexts, and thus ready to be used in everyday clinical practice.

CONCLUSION

Nowadays, there is a broad range of methods for detecting and evaluating executive dysfunction ranging from clinical interview to neuropsychological evaluation. This diversity of methods suggests that current approaches do not account for EF complexity. On many occasions, a dissociation is observed regarding the scores obtained in the tests -many times in normal ranges- and the severe difficulties in daily life. For instance, clinical observation of patients with good performance in traditional neuropsychological tests, but with different degrees of executive dysfunction in real-world functioning (M. Mesulam, 1986).

An innovative strategy is the use of information technology and communication (ICT). The development of technological methods increases ecological validity while keeping satisfactory control of experimental variables (Fan, Dal Monte, & Chang, 2021). In the present work, we propose that SG, and more specifically, the screen-based simulations, offer an opportunity to develop more efficient tools to detect executive dysfunction in everyday life contexts. This statement is supported by five main arguments: 1) SG develops real-world contexts through real simulations based on multimodal interactions, 2) SG are highly interactive environments (natural, social, and interpersonal), 3) SG allow the assessment through ill-structured tasks, 4) SG include advanced game analytics systems, and 5) SG simplify the administration in an evaluation setting.

In conclusion, SG provide meaningful narrative stories and virtual or real environments that immerse the user in natural and social environments with social interactions, while the player needs to adapt his/her behavioral performance to the challenges of ill-structured tasks (natural, social, cognitive, and interpersonal). Moreover, the user has to explore, make decisions, solve tasks, interact socially, interpret the behaviors of others, all of which are interspersed with each other and presented in different forms and sequences. We believe that SG are a well-balanced non-intrusive method for obtaining users’ behavioral data in more naturalistic settings that can be used by clinical personnel as evidence to detect executive dysfunction in real-world performance.