Multipurpose Spatiomotor Capture System for Haptic and Visual Training and Testing in the Blind and Sighted

Abstract

We describe the development of a multipurpose haptic stimulus delivery and spatiomotor recording system with tactile map-overlays for electronic processing This innovative multipurpose spatiomotor capture system will serve a wide range of functions in the training and behavioral assessment of spatial memory and precise motor control for blindness rehabilitation, both for STEM learning and for navigation training and map reading. Capacitive coupling through the map-overlays to the touch-tablet screen below them allows precise recording i) of hand movements during haptic exploration of tactile raised-line images on one tablet and ii) of line-drawing trajectories on the other, for analysis of navigational errors, speed, time elapsed, etc. Thus, this system will provide for the first time in an integrated and automated manner quantitative assessments of the whole ‘perception-cognition-action’ loop – from non-visual exploration strategies, spatial memory, precise spatiomotor control and coordination, drawing performance, and navigation capabilities, as well as of haptic and movement planning and control. The accuracy of memory encoding, in particular, can be assessed by the memory-drawing operation of the capture system. Importantly, this system allows for both remote and in-person operation. Although the focus is on visually impaired populations, the system is designed to equally serve training and assessments in the normally sighted as well.

Introduction

We describe the development of a multifunctional system to provide for effective spatiomotor rehabilitation in blindness and visual impairment. For those who have lost vision, the eye-hand coordination normally available for the manipulation of objects for everyday activities is unavailable and has to be replaced by information from other senses. It becomes crucial to activate cross-modal brain plasticity mechanisms for functional compensation of the visual loss in order to develop robust non-visual mental representations of space and objects. Such nonvisual ‘mental maps’ are needed to guide spatiomotor coordination, reasoning and decision-making. Our multidisciplinary approach to this problem (Likova, 2012; ^{2013; 2016}; 2020) overcomes the shortcomings of traditional rehabilitation training, which can be both tedious and expensive.

To bridge this gap, Likova has developed an effective rehabilitation tool, the Cognitive-Kinesthetic (C-K) training approach to bridge the gap to wide spectrum blind rehabilitation by employing an integral task (drawing) that can affect ‘at one stroke’ a wide vocabulary of core abilities that are building blocks for numerous everyday tasks. For optimal implementation of this form of training, we have developed a multifunctional tactile/kinesthetic stimulation delivery and spatiomotor recording system for the enhancement of spatial memory functions through non-visual stimulation and recording devices. Understanding the behavioral and neural adaptation mechanisms underlying the rehabilitation of vision loss will also meet the broader goal of providing for a well-informed learning approach to functional rehabilitation.

Cognitive-Kinesthetic training

The novel Cognitive-Kinesthetic (C-K) Drawing Method implemented in this system (Likova, 2012, 2013, 2014, 2015, 2018) is based on the spatiomotor task of drawing, because drawing – from artistic to technical – is a ‘real-life’ task that uniquely incorporates diverse aspects of perceptual, cognitive and motor skills, thus activating the full ‘perception-cognition-action loop’. Drawing engages a wide range of spatial manipulation abilities (e.g., spatio-constructional decisions, coordinate transformations, geometric understanding and visualization), together with diverse mental representations of space, conceptual knowledge, motor planning and control mechanisms, working and long-term memory, attentional mechanisms, as well as empathy, emotions and forms of embodied cognition (e.g., Likova, 2012; ²⁰¹³). The Cognitive-Kinesthetic Drawing Method makes it possible to use drawing as a ‘vehicle’ for both training and studying training-based cross-modal plasticity throughout the whole brain, including visual areas activated by non-visual tasks.

The innovative philosophy of this methodology is to develop an array of cognitive mapping and enhanced spatial memory capabilities to provide those with compromised vision to develop – in a fast and enjoyable manner - precise and robust cognitive maps of desired spatial structures independently of a sighted helper; moreover, to develop the ability to use these mental representations for precise motor planning and execution. The trainees learn 1) to access a high-functioning cognitive mapping capability, 2) how to haptically explore tactile maps and encode them in memory to reach a high level of precision and stability of the cognitive maps 3) to store and recall spatial route trajectory information to high precision in the form of a spatially specific cognitive map, 4) to make flexible coordinate transformations, 5) on this basis, to be able to make complex navigational decisions, such as identifying an optimal route by ‘seeing’ and manipulating the cognitive map in in their ‘mind’s-eye’, i.e., on the amodal spatial memory sketchpad (Likova, 2012, 2013), and 6) to recognize and resolve minor departures from the map structure, such as road works and scaffolding obstacles, so as to be able to recover an optimal route when diverted. Note that we use ‘map’ here in the broader sense of any ‘spatial line-image’ in general.

Multipurpose spatiomotor capture system for training and testing of C-K memory & motor control training

This innovative multipurpose capture system will serve a whole range of functions in both the training and behavioral assessment of spatial memory and motor control for both navigation and manual performance. Capacitive coupling through the map-overlays to the touch-tablet screen below them will allow precise recording of i) haptic exploration and ii) drawing trajectories, will facilitate a broad range of in-depth analyses along the full trajectory of learning and rehabilitation.

Thus, this system will ensure unprecedented rigor in the quantitative assessments of spatial memory, drawing performance, and navigation capabilities, as well as of haptic exploration strategies and movement planning and control. The accuracy of memory encoding, in particular, will be assessed by the memory-drawing module on the capture system, which also allows the automated recording of the exploration of the tactile map overlay on one tablet and the drawing of the learned map or route trajectory on the other tablet.

In more detail, the multipurpose spatiomotor capture system consists of two touchscreen tablet computers (e.g., Microsoft Surface Pros) that share a common second display, keyboard and mouse via wireless connections (Fig. 1). The participants sit before the pair of tablets and being blind, visually impaired or blindfolded-sighted, cannot visually process the stimuli, or the exploration or drawing trajectories. Instead, they explore the raised-line image structure with the fingers. The pixel coordinates of the finger and button status on the touchscreen surfaces are recorded at 400 Hz. The left touchscreen is used for the ‘Explore and Memorize’ tasks, where a sheet with a raised-line tactile image is placed over the touchscreen and the time course of exploration of the tactile content is recorded by the tablet. After tactile exploration of the raised-line content, a variety of memory-guided drawing tasks are performed on the right tablet with data similarly recorded. The drawing may be done either with a finger or with an active stylus. The position of the two tablets can also be switched to accommodate participants of different handedness. For the integrated system, the tablet computers sit in recessed cut-outs in an adjustable-angle drawing board which is secured to a table. The recessed rectangles provide a tactile cue for the boundaries of the touchscreens, as well as holding of the tactile sheets in place on the left-hand side. The operator can sit on the same or on the opposite side of the table from the participant and control both tablets via a common keyboard, video monitor, and mouse. Custom software developed for the project provides a graphical user interface allowing for selection of which particular tactile stimuli are being presented as well as for initiation of data acquisition of the participant’s drawing movements.

Figure 1.

Schematic view of the in-person capture system. 1. Tablet ‘Explore’ with a tactile-map raised-line overlay for haptic exploration. 2. Tablet ‘Draw’ to record memory drawing. 3. Operator’s screen. The data are stored on the operator’s computer. The operator can either sit beside the participant for training purposes or across the table for observation during testing sessions.

Data acquisition is started and stopped by the operator with a keystroke on the keyboard. While data are being acquired, an image of the stimulus being either explored or drawn from memory is shown on the operator’s display. Overlaid on the stimulus image, the exploration or drawing data are visualized in another color as they accumulate to give the operator real-time feedback for how well the participant is performing. The haptic exploration, or memory-guided or ‘observational’ drawing data are stored for offline analyses of speed, accuracy and other features of hand-motion trajectories such as exploration strategies and speed-accuracy trade-offs at different stages of the learning process.

Advantages

While the main system description is oriented toward non-sighted application for those without sight or with low vision, the multipurpose capture system is advantageous for use in a wide variety of other applications and populations:

Haptic Rehabilitation Training

A key application of the system is for the haptic training of pictorial recognition and non-visual spatial memory for graphic materials such as those encountered in STEM learning contexts. Images and diagrams converted to raised-line tactile images may be explored and understood by haptic exploration with one or more fingers. In many cases, this is a novel medium for blind users familiar with exploration of objects as three-dimensional structures, as they learn to appreciate how objects such as faces can be represented in two dimensions. Once the images are encoded, the multipurpose capture system provides for the an iterative exploration and drawing procedure, based on the capture of the drawing of the image on the second tablet by the hand that was not involved in the exploration. The switch between hands is an important advantage of the two-tablet system because it enforces the development of an accurate spatial memory of the image structure rather than simply relying on motor memory, as would be possible if done with the same hand.

Supervised/Unsupervised Functionality

Another advantage of the system is its use in either supervised or unsupervised modes. In the supervised mode, the operator sits adjacent to the participant and guides them through the procedures, based on the principles of the Likova Cognitive-Kinesthetic Training methodology. Briefly, these involve activating the perceptual-cognitive-motor loop through an elaborate supervised training process. In the unsupervised mode, the participant works directly with the system after brief initial instructions on its operation. An automated feedback is this mode is provided by the Computerized Recognizability Index (CRI), developed in our lab, which compares the drawn motor configuration with the initial line-image and provides a measure of its accuracy (after affine transformations are taken into account). Its criteria are slightly different from human assessments of the accuracy, but it provides a quantitative index of improvement to guide the participant towards perfect accuracy.

Remote operation

Based on the requirements of the COVID-19 pandemic, which made it impossible to work with human subjects in the lab, it is often necessary to conduct such training under remote operational conditions. There are many other situations that also greatly benefit from a remote operation, such as when participants experience difficulties to travel every day during the 5-day Cognitive-Kinesthetic Training. A remote version of the system has therefore been developed to allow the training station to be installed in the participant’s domicile while the operator has remote access for the supervised training via an internet connection, as diagrammed in Fig. 2. This configuration requires a third tablet computer with a camera to allow the operator to communicate verbally with the participant and observe the participant’s movements and facial expressions to be able to provide the optimal feedback for effective training. The camera view needs to be wide angle for this purpose, so the standard camera view is enhanced by the addition of a fisheye lens mounted in front of the computer camera.

Figure 2.

Schematic view of the remote three-tablet online architecture. 1: Explore computer. 2: Draw computer. 3: Communication computer. 4: Host computer. 5: Wifi hotspot. The data are saved on computers 1 and 2 for transmission and storage on the host computer.

The information about the participant’s activities from all three sources is transferred by internet connections to the host computer or a server for storage and analysis. The control of the three-tablet system at the participant’s station is managed by a remote control application on the host computer, and the two-way interchange of verbal and visual information between the operator and the participant is provided by a virtual communication application, such as Zoom.

The system has been constructed on an adjustable table-top drawing table housing side-by-side Microsoft Surface tablet computers, which are conveniently programmable for the presentation and recording of the navigation learning and drawing of the learned trajectories for subsequent trajectory analysis. The experimenter’s control is facilitated by the integration of a third monitor into the system for real-time monitoring of the output trajectories

Visual Training of Spatial Memory

A further application of the two tablet system is for visually-guided training in the same manner as the haptic/motor training, implementing a visual version of the Likova Cognitive-Kinesthetic Drawing Training. In this case, the procedure can all be carried out on a single tablet computer (with a second one for the remote monitoring and training interchanges if being conducted remotely). STEM or art images or maps can be presented visually on the screen and explored visually rather than tactilely, and can be removed to show a blank screen for the memory-guided drawing phases. The drawing can be done either entirely from memory with no visual feedback, or in the manner of a conventional drawing in which the image appears progressively as it is being drawn, providing continuous feedback of the drawing result to be compared with the internal memory of what has to be drawn. Although conventional, this approach should be less effective at training accurate spatial memory than the approach with no immediate feedback, guiding the entire drawing trajectory from memory according to principles of the Cognitive-Kinesthetic Training, then comparing the finished result with the original to provide global feedback about its success. In this way, the vividness and practical applicability of spatial memory can be maximally enhanced in only a short period of training.

Conclusion

Our Multipurpose Spatiomotor Capture System for Haptic and Visual Training and Testing in the Blind and the Sighted is a powerful novel conceptualization and a tool for both research and applied purposes, such as neurorehabilitation or enhancement of learning and memory. It makes it possible to implement advanced training procedures, such as the unique Cognitive-Kinesthetic drawing and spatial memory training; and, moreover, to implement it both in-person and in a remote mode of operation in a wide range of populations – from the totally blind to the fully sighted.

Figure 3.

Schematic front (left panel) and side (right panel) views of the Remote Three-Tablet online capture system. 1: Raised-line-image explore computer. 2: Drawing tablet computer. 3: Communication computer. 4: Fisheye lens. 5: Drafting table. 6: Wifi hotspot. 7: Participant’s chair. 8: Raised-line image holder. 9: Lighting stand. The data are temporarily stored on computers 1 and 2. The participant’s online location is remote from the operator’s site.

Author Biographies

Lora Likova, Ph.D., has a multidisciplinary background that encompasses studies in cognitive neuroscience and computer science, with patents in the field of magnetic physics and neurorehabilitation, and many years of experience in brain imaging, brain plasticity, human vision, and neurorehabilitation research. She is the Director of the Brain Plasticity, Learning & Neurorehabilitation Lab at Smith-Kettlewell Eye Research Institute.

Kristyo Mineff, M.Eng. is an engineer and a former TV host, who has served as a Director and Board Member of several engineering companies. Over the last decade he devoted his time to implementing a wide range of brain imaging methodologies to advanced applications in neuroplasticity and blindness rehabilitation. He is particularly focused on unravelling the principles underlying the function and architecture of complex dynamic systems such as the human brain.

Christopher Tyler, Ph.D., D.Sc., received his BA in psychology from the University of Leicester (1966), his MSc in applied psychology from The University of Aston (1967) and his PhD and DSc in neurocommunication from the University of Keele (1970 and 2004). Since 1970 he has pursued research in visual neuroscience at Northeastern University, University of Bristol, Bell Laboratories, Smith-Kettlewell Eye Research Institute, and City University of London, together with studies in Renaissance art history and cartography.