Kinect Project: People with dementia or mild cognitive impairment learning to play group motion-based games

Abstract

Introduction

Motion-based technologies (MBTs) could provide nonpharmacologic interventions for people with cognitive impairment (PCI; e.g., dementia or mild cognitive impairment). This study examined the use of errorless learning techniques to empower PCI to use MBT.

Methods

Thirty-eight PCI were recruited to a 10-week (20 × 1-hour sessions) Xbox Kinect bowling group. Video recorded data from first, middle, and final sessions were coded to track (1) number of prompts per turn, (2) independent turn completion, and (3) duration of turns. These values were compared using repeated measures analysis of variance.

Results

Learning and improvement over time was demonstrated in 23 participants who were available for final analysis by significant decreases in number of prompts per turn and turn duration and significant increases in turns completed independently.

Discussion

Errorless learning supported PCI to learn MBT and improve over time as evidenced by their need for fewer prompts, shorter turns, and more turns completed independently, confirming the potential of MBT to provide leisure activities for PCI.

Highlights

• •People with dementia or MCI can learn to use MBTs through errorless learning.
• •Longitudinal application of errorless learning techniques changes over time.
• •Results challenge stereotypes about learning abilities of people with dementia/MCI.
• •Motion-based technology is a feasible group activity for people with dementia or MCI.

1. Introduction

Dementia affects multiple areas of cognitive function including learning and memory, complex attention, language, perceptual-motor, and social cognition [1]^, [2]. Mild cognitive impairment (MCI) causes mild yet measurable decline in at least one aspect of cognition in excess of age-related norms [1]. People with MCI are at a greater risk of developing dementia [1]. There are currently no disease-modifying therapies for dementia, and with numbers worldwide predicted to reach 75 million by 2030 [3], innovative nonpharmacologic solutions to support people with dementia or MCI (henceforth referred to as people with cognitive impairment, PCI) are urgently needed [4].

Nonpharmacologic activities that PCI can complete at home and those that can be delivered in groups are particularly valued. However, changes in cognition can interfere with continuing current activities and reduce opportunities for others. PCI report difficulty engaging in familiar pastimes because of challenges remembering the steps involved in an activity (e.g., dancing) or the characters from television shows [5]. To address these challenges, nonpharmacologic interventions designed to leverage individual's retained skills and abilities can provide enjoyable, meaningful activities. For example, the Computer Interactive Reminiscence and Conversation Aid, a technology-supported reminiscing activity designed to maximize well-rehearsed long-term memory and minimize reliance on impaired short-term memory can act as a cognitive prosthesis [6]. A group study of the Computer Interactive Reminiscence and Conversation Aid demonstrated improvements in cognition and quality of life in PCI living in care homes, which persisted beyond the intervention [7]. Interacting with the Computer Interactive Reminiscence and Conversation Aid not only benefits PCI, it has also been shown to improve caregiver's perceptions of the people they care for [8] and positively impact caregiving relationships [9]. Increasingly, mainstream technologies such as tablets, smart phones, and game consoles are making new interventions available. The addition of accessibility features (e.g., visual prompts) to mainstream applications such as Solitaire can increase their user-friendliness for people with dementia or MCI [10].

Games such as Solitaire are popular pastimes as they provide opportunities to satisfy basic human needs for autonomy (volition) and competence (affecting the environment for a valued outcome) as described in self-determination theory (SDT) [11]. SDT is a theory of motivation concerned with supporting the natural or intrinsic tendencies of human beings to behave in effective and healthy ways [11]. When played with others, games can also provide opportunities for relatedness (feeling connected to others); the third basic human need identified in SDT [11]. In their daily lives PCI experience reduced opportunities for satisfying these three needs and require support to identify and engage in satisfying activities.

Motion-based technologies (MBTs; e.g., Xbox Kinect) can provide varied activities for PCI to play independently or with others [12]. MBTs interpret human motions (e.g., stepping) as commands and elicit corresponding actions in the virtual world (e.g., game character walking). The physical actions required to interact with MBT are intuitive and resemble everyday gestures, particularly when combined with the simplicity of interaction and immediate screen feedback. However, to deliver these benefits requires a good understanding of how PCI learn to use MBT and the best strategies for teaching and supporting them to use this type of technology [12].

Errorless learning is an effective approach for teaching new information or skills to PCI. Errorless learning focuses on preventing and reducing the occurrence of errors (i.e., something done that is considered to be incorrect) when learning to prevent errors from being implicitly stored in long-term memory [13]^, [14]. Examples of errorless learning techniques include breaking down tasks into procedural steps, avoiding trial-and-error approaches, gradually reducing assistance over time, immediately correcting errors, modeling task steps through gestures, and providing verbal instructions [13]^, [14]^, [15]. In contrast, errorful learning approaches, which have also been used to teach new information or skills to PCI, encourage learners to guess and generate errors [13]. There is evidence suggesting that although PCI can learn through errorful approaches, they can learn significantly more through errorless approaches [16].

Dove and Astell [17] observed a group of PCI at a day program playing games on an Xbox 360 Kinect where care staff applied errorless learning techniques. The staff possessed both a general understanding of cognitive impairment (CI) and a specific understanding of each person in the group, which enabled them to adapt their teaching techniques to each person's unique needs. However, the clients were using the MBT at the day program before the start of the observational study [17]; thus, further research is required to examine the use of errorless learning approaches for PCI to use MBT.

The purpose of the present study was to examine how PCI learn to use MBT (Xbox Kinect) through errorless learning techniques and how the application of these techniques changes over time. Specific aims of this study were to examine as follows:

• 1.The applicability of errorless learning techniques including verbal instructions, gesture demonstrations, and physical assistance to teach PCI to use MBT.
• 2.
  Whether longitudinal application of errorless learning techniques to teach PCI to use MBT decreases over time, indicated by the following:

  • •
    Being able to play with fewer prompts from the facilitator.
  • •
    Completing more turns independently.
  • •
    Shorter duration of turns (i.e., play becomes quicker).

It was hypothesized that errorless learning techniques would be applicable when teaching PCI to use MBT and that longitudinal application of these techniques would decrease over time.

2. Methods

2.1. Design

A repeated measures design was used, with comparison of performance at three time points (start, middle, and end).

2.2. Ethics

Ethical approval to conduct this study was obtained from Ontario Shores Centre for Mental Health Sciences. Participants were required to attend one of the three recruitment sites and to agree to all terms outlined in the consent form. Consent was obtained from each participant independently or through a nominated substitute decision-maker if the participant lacked capacity to consent.

2.3. Participants

Thirty-eight participants aged between 58 and 93 (mean 75.39 years) were recruited from three community-based adult day programs (Table 1). Participants were screened for CI using the Montreal Cognitive Assessment (MoCA); the MoCA is scored out of 30, with scores below 26 indicating CI [18]. Given that the MoCA is used for screening purposes rather than for diagnostic purposes, it was not possible to determine the type or exact severity of CI experienced by participants. However, the MoCA scores of the 38 participants confirmed that they all experienced some degree of CI (Table 1).

Table 1.

Participant demographics

Demographic variable	Participants recruited (n = 38)	Participants analyzed (n = 23)
Age (y)	Mean 75.39 (range 58–93; SD 9.49)	Mean 74.83 (range 58–92; SD 8.59)
Sex (M/F)	18 M (47.4%); 20 F (52.6%)	13 M (56.5%); 10 F (43.5%)
MoCA score (out of 30)	Mean 12.47 (range 0–25; SD 6.61)	Mean 12.95 (range 0–25; SD 7.0)
Mobility device use	Device user (16/38; 42.1%)	Device user (9/23; 39.1%)
	Cane (4 of 16; 25%)	Cane (4 of 9; 44.4%)
	Walker (10 of 16; 62.5%)	Walker (4 of 9; 44.4%)
	Wheelchair (2 of 16; 12.5%)	Wheelchair (1 of 9; 11.1%)
Highest level of education	Education information obtained (n = 34/38)	Education information obtained (n = 19/23)
	Elementary school (5 of 34; 14.7%)	Grade 9 (1 of 19; 5.3%)
	Grade 9 (1 of 34; 2.9%)	Grade 10 (4 of 19; 21%)
	Grade 10 (6 of 34; 17.6%)	High school (14 of 19; 73.7%)
	High school (18 of 34; 52.9%)
	College diploma (1 of 34; 2.9%)
	University undergraduate degree (3 of 34; 8.9%)
Prior MBT Experience	No (38; 100%)	No (23; 100%)

Abbreviations: MBT, motion-based technology; MoCA, Montreal Cognitive Assessment; SD, standard deviation.

The facilitator was a 25-year-old female with 17 years of education and 3 years of experience conducting research with PCI. The facilitator was trained to lead the Xbox Kinect sessions by observing trained day program staff coach PCI and other age-related challenges to play games on Xbox 360 Kinect [17]. The facilitator then ran a 20-session “practice” group at a separate day program, to rehearse the prompting techniques identified during the observational phase of the study. This helped to ensure that the facilitator knew when and how to give instructions to participants.

2.4. Materials

This study used the Xbox One Kinect [19] given that interaction with this system relies purely on intuitive and naturalistic movements (e.g., waving an arm), resulting in fewer usability issues [12]. The Xbox One Kinect was chosen over the previous “Xbox 360 Kinect” as the latter cannot accommodate seated play or people using mobility devices [17]. The Kinect Sports Rivals bowling game was chosen because of the familiarity of the game and the simplicity of the movements required to play.

Two Sony Handycam HDR-CX405 video cameras were used to record the sessions. The Observer XT video analysis software [20] was used to code the video recordings for later statistical analysis, using v24 of the Statistical Package for Social Sciences software. A strip of green electrical tape was placed approximately 6 ft away from the Kinect sensor (as per Microsoft's Kinect sensor placement recommendations [21]) to mark a spot on the floor for players to bowl from.

2.5. Environment

The bowling sessions were held in the activity room of each day program, as each room featured a large television or smart board, with ample space for participants to sit between turns. The rooms were configured identically for all sessions. The video cameras were setup at the front and back of the room during the bowling sessions to capture a comprehensive view of the active player, and the facilitator and the whole group. The facilitator was situated next to the game screen during all sessions to assist participants as required.

2.6. Procedure

The Xbox Kinect bowling was offered as a scheduled group activity within each day program twice weekly for 10 weeks (20 sessions per site). Each 1-hour session was led by the facilitator, and the entirety of each session was video recorded. The facilitator initiated each play session by providing a verbal and physical demonstration illustrating how to interact with the technology and play the game by presenting the movements required to play in a stepwise manner. After the initial demonstration, participants were invited one by one to take their turn. Each turn comprised one or two “bowls,” depending if all pins were knocked down during the first shot (i.e., a strike). Participants continued to take turns for the duration of each session.

During each session, the facilitator coached and supported participants to use MBT using errorless learning techniques [14]^, [22], which included verbal instructions, gesture demonstrations, and physical assistance. The facilitator provided participants with assistance in a layered approach from least (i.e., verbal prompts) to most intrusive (i.e., physical assistance), to support each participant's specific needs. In other words, during each participant's turn, the facilitator would observe the active player's facial expressions and body movements; this was used to identify signs of confusion or signs that the participant was about to make an error. The facilitator would then assist the active player using a verbal, gesture, and/or physical assistance prompt. In some cases, different prompts were provided simultaneously (e.g., verbal and gesture).

2.7. Coding scheme

2.7.1. Prompts required

Prompts were classified into three broad categories: verbal prompts (i.e., words used to provide instruction), gesture prompts (i.e., steps modeled using physical actions), and physical assistance (i.e., physical intervention, such as hand-over-hand guidance). Although the number of errors made by participants was not tracked, prompts were provided in response to an error, or when a participant was about to make an error. If different prompts were provided simultaneously (e.g., verbal and gesture prompts), they were coded separately (i.e., two prompts vs one).

2.7.2. Independent turn completion

If a participant was able to complete their turn with no prompts from the facilitator, the turn was marked as being completed independently. However, if a participant required assistance (i.e., prompts) from the facilitator on at least one occasion during their bowling shot(s), the turn was marked as being completed with support.

2.7.3. Turn duration

Participants' turns were timed from the moment they reached the green line and completed when the participants' last bowling ball resulted in a gameplay outcome. If the first bowling shot resulted in a strike, this signaled the end of that turn.

2.8. Data analysis

Video recordings were analyzed by two independent raters (C.H. and S.A.). Each rater underwent training using the Observer XT video analysis software [20] and coding scheme, which included completing an introductory coding exercise, reviewing the videos, familiarizing themselves with the coding scheme and its definitions, and practicing coding a selection of the videos. Before formally coding the videos, inter-rater reliability was assessed [23]. When inter-rater reliability reached at least 80% agreement for each site, the two raters coded the videos for analysis. Coding of the videos included marking each time the facilitator provided a prompt to a participant (and the type of prompt), each time an independent turn was completed, and each time a participants' turn started and ended. This involved capturing frequency and duration data.

Six complete recordings were selected for analysis from each site: the first two sessions (1 and 2), the middle two sessions (11 and 12), and the final two sessions (19 and 20). The two raters coded the entirety of each selected session, with each turn of each participant analyzed. Data from the first two sessions were combined as scores for “start” (T1), the two middle sessions as “middle” (T2), and the last two sessions as “end” (T3). Given the laborious and time-consuming nature of video analysis, three firm time points were used to measure participants' performance over time.

Intrarater reliability was performed to confirm the internal consistency of each coder [24]. The values of κ for three sessions (one per site), coded by the same rater (six sessions in total; three per coder), were calculated as 0.79, 0.70, and 0.87 (C.H.), and 0.97, 0.88, and 0.91 (S.A.), indicating intrarater reliability from substantial to almost perfect agreement [24]. After coding, data from the three sites were combined to extract the overall mean number (and type) of prompts, mean percentage of independent turns, and mean duration of turns. These mean values were then compared across the three time points using a repeated measures analysis of variance (ANOVA) with post hoc analysis (paired t tests with a Bonferroni correction). Statistical analyses were conducted using a P value of <.05.

3. Results

Twenty-three (60.5%) of the 38 participants enrolled in the study (Table 1) were captured on video during the initial, midpoint, and final sessions. As such, statistical analyses were only performed on data from these 23 participants (Table 1). Lost data were accounted for by participants missing a session(s) because of illness, appointments, or general absenteeism, all of which are common in adult day program settings. To further understand the relationship between cognitive ability and learning, the 23 participants were later stratified into two groups based on their MoCA scores (Table 2).

Table 2.

Stratified MoCA analysis

Subgroups	MoCA scores mean (SD)	Prompts mean (SD)			Independent play mean (SD)			Turn duration (s) Mean (SD)
		T1	T2	T3	T1	T2	T3	T1	T2	T3
High MoCA (n = 14)	14–25 (17.85)	6.17 (4.44)	3.25 (3.51)	0.94 (1.03)	16.55 (25.72)	32.1 (30.72)	67.69 (25.19)	34.46 (8.01)	30.55 (8.52)	26.79 (4.9)
Low MoCA (n = 9)	0–11 (5.9)	14.54 (3.97)	7.25 (9.18)	5.65 (8.49)	2.22 (6.67)	11.11 (15.37)	33.2 (29.7)	52.65 (12.92)	36.57 (11.25)	34.06 (9.24)

Abbreviations: MoCA, Montreal Cognitive Assessment; SD, standard deviation.

3.1. Prompts

Three types of prompts were delivered by the facilitator, all of which reduced over time (Fig. 1A). Unsurprisingly, the least intrusive verbal prompts were used most frequently by the facilitator. However, certain participants responded better to gesture than verbal prompts, potentially explaining why the number of verbal and gesture prompts provided became more evenly distributed over time. Physical prompts were used least, which remained consistent over time (Fig. 1).

Fig. 1.

(A) Frequency and type of prompts provided at each time point. (B) Mean number of prompts required (per participant, per turn).

Although the use of prompts was never eliminated, the total number of prompts required by participants significantly decreased over time (Fig. 1B; F = 18.744; P < .0001). Post hoc analysis revealed that participants required fewer prompts at the midpoint (mean 4.82; range 0–136) relative to the start (mean 9.44; range 1–132; P < .01). This continued to drop at the end (mean 2.78; range 0–66; P < .0001). In addition, the mean number of prompts between the midpoint (mean 4.82; range 0–136) and the end (mean 2.78; range 0–66) dropped significantly (P < .05), indicating that participants continued to improve over time.

3.2. Independent turn completion

The number of independent turns was also examined (Fig. 2). During the initial sessions, only seven of 23 participants (30.4%) completed a turn independently, which increased to 15 of 23 participants (65.2%) by the midpoint. During the final sessions, 20 of 23 participants (87%) completed at least one turn independently. Of these 20 participants, seven (35%) completed 75% or more turns independently, and three (15%) completed an entire session independently.

Fig. 2.

Percentage of turns completed independently.

Repeated measures ANOVA revealed a significant difference between the three time points (Fig. 2; F = 26.187; P < .0001), that is, the percentage of turns independently completed by participants significantly increased between the start (mean 10.95; range 0–83.3) and midpoint (mean 23.9; range 0–100; P < .05) and start and end (mean 54.19; range 0–100; P < .00001). Participants also continued to improve between the midpoint (mean 23.9; range 0–100) and end (mean 54.19; range 0–100; P < .01).

3.3. Turn duration

As participants required fewer prompts (Fig. 1B) and were able to play more independently (Fig. 2), the mean duration of their turns also decreased over the three time points (Fig. 3). For example, when comparing the start to the end, six of 23 participants (26.1%) reduced the average duration of turn by 1 second or more, two of 23 participants (8.7%) by 5 seconds or more, three of 23 participants (13%) by 10 seconds or more, and 10 of 23 participants (43.5%) by 15 seconds or more.

Fig. 3.

Mean turn duration (in seconds).

Repeated measures ANOVA revealed a significant difference between the three time points (F = 20.312; P < .0001). Post hoc analysis revealed a significant difference in the mean duration of participants' bowling turns between the start (mean 41.57; range 11.6–135.9) and midpoint (mean 32.91; range 9–100.2; P < .01) and start and end (mean 29.63; range 10.7–82.7; P < .0001). Although mean duration continued to drop between midpoint and final sessions (Fig. 3), this difference was not statistically significant (P = .165).

3.4. Learning versus cognitive ability

To further understand the relationship between cognitive ability and learning, all 23 participants were divided into two groups based on their MoCA scores (Table 2), with the “High MoCA” group being defined as those scoring between 13 and 25 (n = 14) and the “Low MoCA” group defined as those scoring between 0 and 12 (n = 9). These data confirm that the Low MoCA group required more prompts (P < .0001), had fewer independent turns (P < .065), and took longer turns at baseline (P < .003) than the High MoCA group. This continued at all three time points (Table 2). However, analysis of the data from the two subgroups also revealed that the number of prompts required by participants in both groups reduced significantly between the start (T1) and the end (T3) of the Xbox bowling program (High MoCA, P < .001; Low MoCA, P < .006). Similar analysis of the number of independent turns also showed that these increased significantly in both groups between the start and the end of the Xbox bowling program (High MoCA, P < .003; Low MoCA, P < .013). Finally, comparison of turn duration indicated that both groups became significantly quicker at taking their turns by the end of the Xbox bowling program (High MoCA, P < .001; Low MoCA, P < .0001).

4. Discussion

This study demonstrated that PCI can learn to use MBT through errorless learning approaches, and the application of these techniques reduces over time. Participants significantly improved on three indictors of learning: (1) use of prompts, (2) number of independent turns, and (3) duration of turns. Significant improvements were found on all three indicators by the midpoint sessions and continued to improve until the final sessions. These findings corroborate previous research demonstrating spared procedural learning in PCI [25]. Of importance for PCI, their ability to learn to play Xbox bowling and keep improving can be seen to fulfill their needs for autonomy and competence, as outlined in SDT [11]. Essentially, bowling on Xbox Kinect provides PCI an opportunity to act in their environment (autonomy) and succeed in a fulfilling activity (competence), when participating in a group (relatedness). When these three innate needs are satisfied, it is proposed that people experience enhanced self-motivation and improved social and mental health, which are important for supporting overall well-being [11].

These findings confirm the potential of MBT activities as nonpharmacologic interventions that empower PCI to live well. The group model used here provides opportunities for social interaction, and the experience of conducting the study in day programs as part of their regular activity schedule demonstrates the potential for implementation. This is one of the first studies examining the use of MBTs by PCI for leisure activities and further investigation of the impacts (e.g., psychosocial, cognitive, and physical) is indicated [26].

This study specifically examined tailored techniques to teach and support PCI to use MBT. Errorless learning [13]^, [14] via verbal instructions, gesture demonstrations, and physical assistance successfully empowered PCI to learn to play Xbox Kinect bowling, maintain this learning, and improve over time. Of these three prompt types, verbal–the least intrusive–were most used, with physical assistance accounting for only 19 of 2563 total prompts across the whole study. This information can inform development of errorless learning techniques to empower PCI to adopt other technological and nontechnological innovations.

These findings have implications regarding the ways in which people with ranging severities of CI can learn through errorless approaches. Furthermore, these findings can be used to inform the ways care staff can be trained to teach and support PCI to use MBT (e.g., showing them what types of prompts to use and when, and how to tailor approaches for different individuals). Particularly, the facilitator must be sensitive to each client's cognitive, physical, and social needs. This informs the selection and delivery of prompts and must be kept under constant review to adjust prompts across sessions. For example, when providing gesture prompts in the present study, some participants would focus their attention on the facilitator, rather than the game screen. Knowing this allowed the facilitator to adjust the type of prompts used (e.g., avoid using gesture prompts) to maximize participant success and enjoyment.

The present study features several strengths. Although previous studies highlight the feasibility and benefits of using MBT with older adults [27]^, [28], this study focused specifically on PCI. In addition, the present study quantified the use of errorless learning when coaching PCI to use MBT through analysis of video recorded data. This approach to data collection and analysis has never been applied to the use of MBT with PCI, adding novelty to the current literature. The heterogeneity of the sample is another strength, which demonstrates the ability of PCI with different levels of cognitive ability to learn to use MBT regardless of age, sex, or type of CI. This finding was confirmed through stratified analysis of MoCA scores, which showed that although participants in the Low MoCA group required more prompts, completed fewer independent turns, and took longer to complete their turns than the High MoCA group, they still showed a significant improvement in performance, which was sustained over time.

This study has some limitations: First, the heterogeneity of the sample can be considered as a limitation as precise information about the participants' diagnosis was not directly available at the day programs. This is a common situation in day programs offering services to all PCI regardless of type or severity of CI. In this real-world care setting, the results of the study highlight the potential of MBT as a group activity for all clients rather than a specific subset (e.g., PCI of specific types). Second, the present study did not compare the efficacy of errorless learning with other learning approaches (e.g., errorful learning). However, given the improvement in playing, the effectiveness of errorless learning in teaching PCI to use MBT is clear. Third, the within-participants design meant that there was no control group. Finally, although the facilitator was trained to instruct participants, prompting became easier over time as the facilitator learned how individual participants would respond to different prompts, which may have impacted the results. Future studies could compare the efficacy of novice versus trained facilitators, particularly staff in day programs who can implement MBTs as an activity for their clients.

5. Conclusion

PCI can learn to use MBT through errorless learning approaches, with the application of these techniques changing over time. These findings challenge negative stereotypes about the abilities of PCI to learn while supporting the feasibility of using MBT as a group activity in adult day programs. Future work should focus on developing accessible activities on MBTs tailored for PCI [29]. This could provide scalable satisfying nonpharmacologic interventions for PCI.

Research in context.

• 1.Systematic review: The authors reviewed the literature using traditional (e.g., PubMed) sources, as well as meeting abstracts and presentations. Recent publications relating to cognitive impairment and motion-based technology (MBT) interventions were identified. These relevant sources are appropriately cited.
• 2.Interpretation: This article describes how people with dementia or MCI can learn to use MBT (Xbox Kinect bowling) using errorless learning techniques. Findings show that people with dementia improve over time through the longitudinal application of errorless learning techniques.
• 3.Future directions: To maximize the benefits of this type of technology for people with dementia or MCI, further work should explore the use of MBT in other care contexts (e.g., long-term care homes), as well as the development of bespoke motion-based games that are both engaging and accessible for people with a range of cognitive and physical abilities.