Interdisciplinary Concepts for Design and Implementation of Mixed Reality Interactive Neurorehabilitation Systems for Stroke

Abstract

Interactive neurorehabilitation (INR) systems provide therapy that can evaluate and deliver feedback on a patient's movement computationally. There are currently many approaches to INR design and implementation, without a clear indication of which methods to utilize best. This article presents key interactive computing, motor learning, and media arts concepts utilized by an interdisciplinary group to develop adaptive, mixed reality INR systems for upper extremity therapy of patients with stroke. Two INR systems are used as examples to show how the concepts can be applied within: (1) a small-scale INR clinical study that achieved integrated improvement of movement quality and functionality through continuously supervised therapy and (2) a pilot study that achieved improvement of clinical scores with minimal supervision. The notion is proposed that some of the successful approaches developed and tested within these systems can form the basis of a scalable design methodology for other INR systems. A coherent approach to INR design is needed to facilitate the use of the systems by physical therapists, increase the number of successful INR studies, and generate rich clinical data that can inform the development of best practices for use of INR in physical therapy.

Interactive neurorehabilitation (INR) systems track patient movement and provide adaptable feedback based on evaluation of movement performance¹ for sensorimotor rehabilitation.^2–5 Interactive neurorehabilitation systems for stroke rehabilitation have led to practice-dependent improvement in motor function of the affected arm⁶ and have contributed to greater improvements in limb function in comparison with conventional therapy alone,⁷ although the extent to which INR is more effective than traditional therapy is still under investigation. These systems can engage different levels of therapist supervision, such as in the home, where supervision is reduced.^8,9 Interactive neurorehabilitation also can vary based on the inclusion of robotic devices,^10–12 virtual reality environments,¹³ or mixed reality (MR).¹⁴

Mixed reality INR, which integrates virtual environments¹³ with physical objects to manipulate or navigate, has the potential to help patients focus on self-assessment and facilitate training that can transfer to other contexts¹⁴ such as activities of daily living (ADL). Increasing the amount of digital feedback dissociates the patient from the physical task by changing the context in which it is performed, whereas decreasing or eliminating the presence of digital feedback requires the patient to complete the task more independently. Dynamically adapting the amount of digital feedback helps the patient connect learning in the virtual domain to physical action. Although some dissociation is beneficial for engagement and reducing frustration, pure VR INR can impede transfer of gains to ADL in the physical world.^14–16 Transference of gains to ADL also can be limited by training movements that do not directly translate to daily, functional tasks.

Although INR is in a relatively early stage of development with many unknowns, we propose that interdisciplinary knowledge has much to offer when merged with neurorehabilitation and physical therapy knowledge. The arts, for centuries, have studied and constructed complex displays for context-aware self-reflection.¹⁷ Learning through creative practice has formed the basis of constructivist learning methods^18,19 that are prevalent in 21st century mediated learning. Rapidly evolving applications of interactive media (from mobile apps to interactive data visualizations) also rely heavily on the integration of arts, computing, and mediated learning knowledge.²⁰

Our experience with the development and testing of 2 MR INR systems—the adaptive mixed reality rehabilitation (AMRR) system and the home-based adaptive mixed reality rehabilitation (HAMRR) system—demonstrates that the above interdisciplinary knowledge can be applicable to the design and implementation of many components of MR INR. The exact optimal implementation of these interdisciplinary concepts in INR therapy is still unclear, as all key components of INR therapy should be customized to each patient's needs, progress, and training supervision context. Large-scale evidence for how to structure automated adaptive protocols for rehabilitation is currently lacking. The diversity among approaches taken to design and implement INR systems makes the existing body of evidence across studies incomparable.

In this article, we present 4 key MR INR design and implementation concepts learned from our experience designing AMRR and HAMRR systems: (1) use of interdisciplinary knowledge for designing key INR components (including assessments, task objects, and feedback), (2) use of a modular architecture, (3) use of self-imposed constraints for combining components into adaptive INR experiences, and (4) application and evaluation of methods. We feel that these concepts can form a useful first step to begin a dialogue with the physical therapy community to gain more standardized implementation and resulting evidence for adaptive INR.

Overview of Example Systems

The AMRR system, which provides detailed evaluation information and interactive audiovisual feedback on motor elements within the performance of reach to grasp/touch tasks,^1,21 was designed for fully supervised training in the clinic. The system uses 11 infrared motion-sensing cameras to track 14 reflective markers worn by the patient on the arm and torso. Based on the 3-dimensional (3D) location of the markers, 40 key kinematic features are used for computational evaluation of the full arm and torso movement and to generate real-time audio and visual feedback and a post-reach visual summary (video 1 below).

The HAMRR system is an extension of the AMRR system for minimally supervised, long-term use in the clinic and eventually in the home. The HAMRR system was designed to be portable with a reduced sensing infrastructure (Fig. 1), requiring only 4 motion capture cameras, a single marker worn on the patient's wrist, and a small plate with 4 markers worn on the shoulder (Fig. 2), accompanied by increased use of embedded sensing in the task objects (Fig. 3) and patient's chair. Therefore, the movement features captured by the HAMRR system were limited to the higher-level categories of hand spatial and temporal performance, torso compensation, and object manipulation. The HAMRR system was designed to be run by the patient using instructional videos on how to perform tasks and utilize the feedback throughout training (video 2 below). The HAMRR system trains and provides unique feedback for 3 different levels of activity, referred to as interaction levels: (1) concurrent and summary feedback per single task (supported and against-gravity reach to touch, grasp, and lift tasks), (2) summary feedback per set of repetitive tasks, and (3) summary feedback for complex tasks (transporting an object between 2 locations) (Fig. 4; video 3 below).

Figure 1.

The home-based adaptive mixed reality rehabilitation (HAMRR) system comprises 2 main components: a media stand and a table. The media stand supports 4 mounted motion capture cameras and a computer that performs all of the motion analysis computation and presents all of the feedback. A custom-built table accommodates customized touch and grasp, affording objects for training reaches in different orientations. A custom-built chair houses pressure sensors on the back to sense applied pressure.

Figure 2.

The marker configuration for home-based adaptive mixed reality rehabilitation (HAMRR). A collection of 4 markers is worn near the left shoulder, and a single marker, on an elastic wristband, is worn on the wrist.

Figure 3.

The collection of task objects (from left to right): flat, planar object for fully supported reach to touch; button for slightly elevated reach to touch; cone for fully supported reach to grasp; cylinder for fully supported reach to grasp with larger hand aperture; and cylinder on an elevated platform for reach to grasp against gravity. The cylinder object was used for transportation tasks, in which it was moved between 2 locations on or elevated off of the table.

Figure 4.

The multiple levels of feedback in home-based adaptive mixed reality rehabilitation (HAMRR) within the narrative of traveling on a boat. Level 1 shows post-reach trajectory error though the shape and color of a stone path leading to a boat. Level 2 provides qualitative summary feedback on the overall performance of a group of repetitive reaches through distortions in the shape of a boat. Level 3 provides qualitative feedback on the overall performance of a multiphase or transport task through a short animation in which the task performance is conveyed in the visual quality of the boat's travel.

Four MR INR Concepts Arising From Interdisciplinary Design and Implementation of AMRR and HAMRR Systems

Use of Interdisciplinary Knowledge for Designing Key INR Components

The following information summarizes key components of our systems and how we developed them based on application of interdisciplinary expertise.

Integrative assessment using kinematics, therapist ratings, and clinical measures.

The AMRR and HAMRR systems integrate complementary methods for assessment of movement performance. Clinical scales provide a validated means for high-level evaluation of functionality. Kinematics offer detailed and accurate assessments of individual motor components (eg, hand trajectory accuracy, speed) across multiple timescales. In practice, kinematic assessments can assist the therapist in adapting training and ultimately provide a means for automated adaptation. The AMRR system introduced a composite measure (Kinetic Impairment Measure)²² that integrated multiple reach and grasp kinematic features. This impairment measure aligned with Wolf Motor Function Test²³ (WMFT) scores, demonstrating that low-level measures of movement quality and high-level measures of functionality can both improve following training with AMRR.²⁴ Although the assessment of more complex tasks trained in HAMRR (eg, transporting an object) also can be informed by kinematics, such multiphase movements may be influenced by several movement components with unknown weights of contribution and are subject to high variation across movers.²⁵ Preliminary movement quality classifiers (based on higher-level spatial and temporal kinematic elements such as path shape and velocity profile) have been created for kinematics-based assessment of such complex tasks.^26,27 However, we propose that the therapist's extensive experience in modifying procedures to optimally challenge the patient can provide insight into how individual aspects of performance should be weighted when constructing higher-level assessments of complex movements. Thus, a video rating iPad tool was created to capture therapist ratings of patient movement within HAMRR across various performance categories. The resultant ratings and kinematics have been used to improve sensitivities of such movement classifiers and to explore semiautomated assessment tools.²⁸

Variety of smart objects for training.

The AMRR and HAMRR systems have a wide collection of smart objects that can sense the patient's manipulation during training. These objects can be used for a variety of tasks (from reach to touch or grasp to transportation) by a wide variety of patients (varying in hand aperture and arm impairment) (Fig. 3). The HAMRR system's objects could be placed in 3 locations across the table for both supported and against-gravity reaches and in multiple combinations of training. For example, a task could require that 2 objects need to be touched in sequence as a training precursor to a transportation task.

Multistream feedback using modality appropriateness.

The AMRR and HAMRR systems provide multiple streams of feedback on the performance of functional tasks, where a stream is a single flow of continuous information controlled by one aspect of movement. The feedback design applies modality appropriateness,²⁹ which refers to the extent to which a specific sensory modality provides the most accurate or appropriate information for each specific movement component.³⁰ For example, visual feedback, which can provide guidance for correcting trajectory errors in goal-directed arm movement,³¹ is used for communicating spatial information in AMRR^1,16 and HAMRR (video 3). Audio feedback is well suited for communicating temporal knowledge³² (video 4 below) and movement patterns requiring complex timing or synchronization,^33,34 such as using a musical rhythm to communicate the hand's acceleration and deceleration in reaching to a target. Compensation is communicated using a discrete sonic indicator (eg, brief sound with a distinct sonic identity) in AMRR and HAMRR, which can be easily overlaid with other more prominent streams of feedback to facilitate associations among different aspects of movement, such as the occurrence of compensation with trajectory error. To integrate multiple feedback streams in a complementary manner, the feedback design uses arts composition concepts, which provide well-developed methods for effective display of complex multistream information (eg, the use of film theory to integrate complex visuals and music³⁵). The AMRR and HAMRR systems use visuals to concurrently communicate the explicit task goal (recompile a fragmented picture by completing a task in AMRR) and music to induce an effective response and sense of progression (eg, musical feedback produces a “mood” [eg, hurried, jerky, smooth] to communicate quality of performance of a complex task in HAMRR). More examples are discussed in detail elsewhere.^1,21

Feedback that ranges from prescriptive, to assistive, to suggestive.

Although prescriptive feedback (eg, instructional information that directly represents the patient's physical experience) can help learning in the early stages of therapy, the majority of feedback for motor learning should not be prescriptive, thus promoting self-assessment as the user is encouraged to develop independent movement strategies.^36,37 In contrast to direct representation, abstract representation uses audio or visual metaphors to represent the patient's action (video 1, Fig. 4), casting the problem in a different context that requires interpretation by the patient. The AMRR and HAMRR systems use abstract auditory and visual representations that communicate explicit aspects of performance, such as magnitude and direction of error, for the patient to correct his or her movement (eg, in HAMRR, a path or rocks leading to a boat moves to the right when the hand deviates far right from an efficient path). The AMRR and HAMRR systems also provide suggestive feedback to promote more implicit understanding achieved through exploration and self-discovery, often requiring multiple interactions with the feedback. Suggestive feedback is intended to facilitate more independent learning and self-evaluation and may be used to summarize performance over multiple reaches. This concept is demonstrated as the display of a well-formed versus broken boat in HAMRR, in which the nature of the boat reflects overall spatial performance across 5 reaches (Fig. 4).

Varied action-feedback relations, ranging from real-time to terminal and summary.

The time delivery of feedback provided by AMRR and HAMRR varies based on a movement component's relevance to performance and the desired amount of patient self-evaluation. Aspects of movement that are continuously monitored by the mover while performing an action, such as end-point spatial progress toward a target,³¹ may require continuously delivered concurrent feedback for detailed knowledge to correct error (eg, continuous particle movement in video 1). Intermittent feedback (eg, brief on/off audio indicators) is provided on aspects of movement irregularly relevant to performance,³⁸ such as torso compensation during a reaching activity, so as to not interfere with the continuous monitoring of the end effector. Terminal feedback is provided immediately following an action, and summary feedback is provided after multiple actions are completed.

Interaction strategies that connect component learning to complex task execution.

Using AMRR, the therapist can adjust the number and type of feedback streams and the frequency of feedback to continuously engage and challenge the patient. The therapist can help contextualize the feedback for the patient and shift focus from the execution of an individual motor element to the performance of the full task. To utilize some of these learning strategies during unsupervised therapy, feedback in the HAMRR system was organized into a 3-level reductive hierarchy, defined in this context as a compositional technique for structured learning¹⁸ in which feedback on low-level aspects of movement are explicit and detailed, while feedback on more abstract, high-level measures of performance are summarized. For example, in low-level training, fundamental aspects of the media (eg, color, shape) are prominently used to explicitly communicate the magnitude and direction of trajectory error for a single reach. In mid-level training across a set of reaches, the feedback is provided in the context of a narrative metaphor (eg, a broken boat with clashing color to demonstrate inefficient performance) (video 3). In high-level training, the quality of performance of a complex transportation task is communicated through the weather elements of a boat trip and the mood of the accompanying music. The basic color, shape, and rhythmic elements established at the low level are held consistent in higher levels but are embedded in more complex scenes and narratives. The overall exposure to different interaction levels is continuously mixed and in various sequences (high level to low level, or low level to high level) to help refresh the patient and facilitate connections of low-level movement components with high-level movement quality (Fig. 5).

Figure 5.

Each week was composed of 3 sessions. Each session had a training focus including: manipulation, trajectory, speed, compensation, and transportation. As therapy progresses, increased time is spent on complex tasks. The progression does not occur in one direction. For example, more real-time feedback was used in session 11 when compensation was introduced, before reducing real-time feedback in session 12.

Use of a Modular Architecture

The following key features are used in our systems to help adapt training to various patient needs.

Architecture changes based on therapy context.

Because learning occurs within highly individualized contexts (eg, each individual constructs his or her own knowledge from first-person experience), the path to achieving knowledge can be nonlinear and vary across learners. The accumulation of experiences contributes to a complex internal model that continuously evolves with more experience.³⁹ Motor learning and other statistical learning⁴⁰ research suggests that internal models are formed based on a collection of perceived rules, associations, and probabilities that are distilled from a variety of experiences in which people physically engage with their environment.⁴¹

The development of the AMRR and HAMRR systems demonstrates that one system cannot encompass all of the necessities for learning across environments, but rather systems should facilitate a continuous experience. The AMRR system utilized more sensing infrastructure because the system was permanently installed in a clinical space. The training protocol was generated by the therapist as the therapist observed each session.^24,42 In transitioning to unsupervised environments with HAMRR, consideration was given to reducing AMRR's footprint while still maintaining some of the core movement sensing and analysis ability and feedback designs. Therapy protocols were pre-composed with opportunities for periodic adjustments of sensitivity by the therapist. To produce a coherent experience across different contexts, AMRR and HAMRR were designed with consistent application of key components described above (videos 1 and 3).

All elements can be adapted.

The AMRR and HAMRR systems have multiple adaptable components to accommodate a range of patient abilities (video 5 below). Tasks can be adapted along dimensions of object manipulation required to complete a task (Fig. 3), location or elevation of the object (Fig. 1), and object transportation between 2 locations (video 3). The sequencing of various task dimensions is another layer of adaptability with which protocols can be created using different sequences and dosages of tasks. Furthermore, based on the training focus, the types of feedback provided can be adjusted, and the corresponding sensitivity of the feedback can be adjusted to vary challenge.

Type and level of supervision can vary.

Interactive neurorehabilitation can be designed for fully or lightly supervised contexts, but the system's interaction should change accordingly. For example, the HAMRR system's experience was designed to fade the amount of supervision from the presence of the therapist at every session to only every third session. Therefore, as previously discussed, HAMRR's feedback structure was designed to gradually facilitate self-evaluation and included instructional videos so patients could self-direct therapy.

Hardware and software components can change in a context-specific manner.

Hardware and software design of both systems vary based on context (ie, physical environmental constraints and available therapist supervision). The AMRR system used sensing hardware that could be permanently installed and full arm and torso marker sets applied to each patient because of the fully supervised clinical space. Designing HAMRR as a lower-cost system for minimally supervised environments required the use of fewer motion capture cameras and markers (Fig. 1) as well as the use of open source,⁴³ 3D printing,⁴⁴ and simple sensor and microcontroller⁴⁵ solutions that are easily extendable and typically cost less. Commercial technologies also may provide low-cost sensing solutions.⁴⁶ The HAMRR system's software architecture introduced a system control component that automatically progresses a patient through a session while providing audiovisual demonstrations and instructions about the tasks and feedback (video 2).

Use of Self-imposed Constraints for Combining Components Into Adaptive INR Experiences

The following constraints, based on interdisciplinary concepts, were imposed at the beginning of the design process of the AMRR and HAMRR systems as rules for combining and testing the implementation of the previously discussed components. The rules are high level, and as such, limited, but are proposed as an initial step in developing more specific and nuanced INR implementation.

Support active learning and engagement.

Efforts to understand underlying mechanisms that foster recovery suggest that active engagement, challenge, and independent discovery can facilitate neuroplasticity.⁴⁷ In stroke rehabilitation, problem solving is critical to enhance motor learning.⁴⁸ Arts-based abstract metaphors provide an exemplary medium for implicitly shaping the individual's experience without explicit instruction.⁴⁹ The arts also provide guidance on achieving long-term engagement over a sequence of separate sessions: consider how each chapter in a novel leads to anticipation of culmination.⁵⁰ The HAMRR experience was designed to last over multiple weeks. Therefore, the feedback was designed within a larger boat narrative to provide a continually aggregating experience across multiple weeks of training.

Balance repetition and variation.

The AMRR and HAMRR systems at their cores support repetitive training, for which the amount of practice has been found to correlate to motor improvement.⁵¹ However, although blocked repetitive tasks may show short-term benefits, long-term generalization and retention are better supported through introducing variability in a training protocol.⁵² Related research on how learning aggregates with repetition and variation is being explored in motor learning,^53,54 machine learning,^55,56 and constructivist learning.^18,19 As AMRR was fully supervised, a therapist could adjust the relative levels of repetition and variation continuously. However, with HAMRR, a first attempt was made to compose fixed protocols that could begin to test the desired amount and sequencing of repetitive training.

Assess both changes in functionality and movement quality.

Both task completion and its quality of performance are evaluated in AMRR and HAMRR. Assessing the functionality of completing the task alone is not sufficient. Other aspects of the movement, such as amount of torso compensation, degree of elbow extension, degree of wrist rotation, and other kinematic measurements within AMRR, help distinguish between recovery of prestroke movement patterns and compensatory movements.⁵⁷

Application and Evaluation of Methods

The design and implementation constraints previously presented were tested within the context of 2 studies.

Study evaluating the application of AMRR.

The goal of AMRR was to focus on improving movement quality in tandem with functionality by giving the therapist the ability to focus on any movement quality features relevant to each patient. As a result, AMRR required sensing of up to 40 kinematic features as well as providing feedback on several of these features in any combination desired.

The AMRR system uses a limited set of repetitive tasks, each incorporating the same key motor components and corresponding feedback to promote aggregate, generalizable learning. Most feedback provided was in real time, while most summary and high-level performance feedback was left to the therapist to provide. For the study protocol, the therapist was provided with implementation rules and some constraints: start from simple tasks and move to complex tasks to support engagement and active learning and reduce frustration; introduce each new component first in a simple context before a complex context; focus first on functional components (task completion) before addressing movement quality components (torso compensation); repeat each set at least twice but no more than 3 times. The therapist was free to choose any overall path that followed these rules to fit a given patient's impairment profile and training needs.

The results from a study with 11 survivors of stroke (1 severe, 3 moderate, 7 mild; median upper extremity Fugl-Meyer Assessment [UE FMA] score=51/66, first-third quartile [Q1–Q3] range=45.23–52), who each used the system for 1 month, showed that kinematic and clinical measurements improved significantly, but assessments of ADL and related measures did not.²⁴ Also, different kinematic measurements improved for each patient in consistent amounts independent of varied training experiences, supporting the use of highly adaptive training using the AMRR system.²⁴ In contrast, a control group of 10 patients with stroke (1 severe, 4 moderate, 5 mild; median UE FMA=52/66, Q1–Q3=44–54) receiving an equal dosage also improved in clinical measurements but not significantly or consistently in kinematics.²⁴ Distributions of the percent change in kinematics were found to be significantly different between groups.²⁴ This finding suggests movement quality was stylized by training fostering active learning through interactive self-assessment and the ability to focus on stylizing performance of motor components, which can be assisted using appropriate feedback.

Study evaluating the application of the HAMRR system.

Although intended for home use, the HAMRR system was first pilot tested in a minimally supervised environment in the clinic to assess the feasibility of training across different testing sites, the ability to replicate some of the experience of therapy strategies seen in the testing of the AMRR system in a less supervised environment, and the ability of new complex tasks to connect to ADL and maintain patient interest more than simple repetitive tasks. The previously discussed constraints were utilized to develop 2 possible protocol paths for 15 days of training: one included less emphasis on transportation tasks against gravity than the other to accommodate patients who are more impaired.

The first 3 of the 15 sessions were observed by the therapist to allow the patient to experience the breadth of training environments and ask questions. The therapist also formed an initial assessment to help determine which of the 2 paths was most appropriate. The remaining 12 sessions were minimally supervised with a system controller present in case of any system issues but not to offer any other assistance with therapy. Before every third session, the therapist video recorded the patient's performance of tasks without feedback using the iPad tool, reviewed selected videos with the patient from previous weeks, and administered a questionnaire.

Each session was limited to 100 reaches, as in AMRR, organized in sets of 5 reaches (defined as a group of sequential reaches with the same task goal and feedback environment) to help minimize fatigue. A given set could be repeated in a sequence only once to adhere to the repetition and variation constraint. Each session balanced time spent on simple and complex movements. For example, during a session, a patient could begin with a simple reach to touch an object and then immediately practice transferring this knowledge to the more complex task of reaching to grasp a cone. Initially, the therapy focused on low-level motor elements and their integration in repetitive tasks. As therapy evolved, patients spent increased time on complex movements, and more feedback was provided about the overall quality of the movement, encouraging the patients to break down that information, through self-assessment, into individual motor elements to correct an error in a complex movement (Fig. 5).

Initial results showed that all 8 patients with stroke (6 mild, 2 moderate) could successfully engage in self-directed therapy with minimal assistance from personnel in a consistent manner across 2 study sites. Use of the HAMRR system led to significant improvements in scores on both the WMFT and the motor section of the FMA⁵⁸ (Table). Significant improvement in clinical measurements indicates that interactive training predominantly focused on self-evaluation of more complex tasks can translate into improvement of untrained complex tasks. Preliminary analyses of the pretest and posttest kinematics indicated inconsistent improvements across patients for the majority of measurements. Because all patients were exposed to a predetermined protocol without continuous adaptation to address specific training needs, inconsistent changes in kinematics may have been partially due to a ceiling effect for the patients with mild stroke, who were comparatively high functioning and may have required more challenge. The significant and more consistent improvement in kinematics in the AMRR study suggest movement quality stylization is possible. However, AMRR's results also indicate the importance of more responsive adaptation than what HAMRR provided. Although the 2 patients with moderate stroke were able to complete therapy, both required some task modifications, such as using objects that required a reduced grasp aperture. Currently under investigation is the potential for correlations among changes in kinematic performance, training paths, and clinical scores to determine which segments of unsupervised training were most effective or need redesign. A full statistical review of the data and potential correlations is in progress.

Table.

Home-Based Adaptive Mixed Reality Rehabilitation Clinical Scale Preliminary Results Measured Before and After Treatment^a

^aPreliminary results of group-level changes in clinical scores (Fugl-Meyer Assessment [FMA] for motor function, Wolf Motor Function Test: Functional Ability [WMFT-FA], Wolf Motor Function Test: Total Time [WMFT-TT], and mean Motor Activity Log [MAL] for quality of movement before and after treatment. W values (sum of signed ranks) and P values for group-level comparisons, using a one-sided Wilcoxon signed rank test (confidence interval=95%), are listed under each posttreatment result.

Remaining Design and Implementation Considerations

Although AMRR and HAMRR studies have provided some confirmation of our proposed MR INR practices, the following are remaining issues that need to be addressed.

Connecting Movement Quality, Functionality, and ADL

Although improved movement quality can coexist with functional improvement, the exact linkage is unknown. For example, the optimal ratio of component training to functional level training is unclear, as is defining which component of training should be given relative priority.

Dosage, Sequence, and Adaptation

The optimal dosage and sequence of tasks and feedback environments, provided in a customizable manner for each patient, is unknown. Protocol customization requires adaptation decisions that predict the progress of training and its potential effect on improving ADL. Automated adaptation through decision networks,^59,60 adaptive training schedules,⁶¹ and more finite, reactive sensitivity parameters^62,63 are promising applications for INR. More field testing of these approaches using MR INR systems that have well-designed components, leverage motor learning and mediated learning rules and concepts, and provide integrative assessment data as discussed is needed.

Feedback Used by Patients

Although the appropriate way to evaluate the use of feedback relies on multiple sources of measurement (eg, questionnaires, changes in performance, active discussion with the patient), the most appropriate sources and combination thereof are unknown. Preliminary evidence from AMRR²¹ revealed strong correlations between improved performance and the preceding presence of relevant feedback but did not account for how the effect changed over time or the cross-effects that may result from using multiple feedback components concurrently. More investigation also is needed to determine how immediate changes in performance following instantaneous feedback relate to long-term changes based on the aggregate effect of the feedback over time. To sample how the patient interacts with the system, the HAMRR protocol included a questionnaire on the ease of use of the system's interface and feedback. Although results indicate that all patients found the system interface and feedback easy to understand and use, these results cannot determine if the patient is actually applying the feedback to correct performance during training. Thus, future extensions of the HAMRR protocol will include interactive probes (periodic questions by the system on how the patient comprehends the feedback) to better evaluate feedback utilization.

Conclusions

Mixed reality INR demonstrates promise as a valuable tool for neurorehabilitation therapy, with the potential to improve with subsequent research. Potential benefits lie in focusing on designing MR INR for long-term home-based therapy emphasizing adaptability and ability to translate training gains to functional improvement. Because a coherent methodology has not yet been established based on studies within the field, design and implementation of INR systems should maximize the benefit of existing interdisciplinary knowledge.