Feasibility and Acceptability of Remote Neuromotor Rehabilitation Interactions Using Social Robot Augmented Telepresence: A Case Study

Abstract

There is a growing need to deliver rehabilitation care to patients remotely. Long term demographic changes, geographic shortages of care providers, and now a global pandemic contribute to this need. Telepresence provides an option for delivering this care. However, telepresence using video and audio alone does not provide an interaction of the same quality as in-person. To bridge this gap, we propose the use of social robot augmented telepresence (SRAT). We have constructed a demonstration SRAT system for upper extremity rehab, in which a humanoid, with a head, body, face, and arms, is attached to a mobile telepresence system, to collaborate with the patient and clinicians as an independent social entity. The humanoid can play games with the patient and demonstrate activities. These activities could be used to perform assessments in support of self-directed rehab and to perform exercises.

In this paper, we present a case series with six subjects who completed interactions with the robot, three subjects who have previously suffered a stroke and three pediatric subjects who are typically developing. Subjects performed a Simon Says activity and a target touch activity in person, using classical telepresence (CT), and using SRAT. Subjects were able to effectively work with the social robot guiding interactions and 5 of 6 rated SRAT better than CT. This study demonstrates the feasibility of SRAT and some of its benefits.

I. INTRODUCTION

Growing shortages of neuromotor rehabilitation clinicians [1], [2] and changing demographics present a challenge to delivering effective rehab care. Frequent rehab care is critical for convalescence from conditions such as stroke and cerebral palsy (CP), which can limit a person’s ability to fully interact with the world around them. One option for overcoming these challenges is to provide rehab care using telepresence, i.e. telerehab, a subcategory of telehealth, the remote delivery of health care using telecommunications [3]. Telehealth has recently seen a large increase in use as a result of the COVID-19 pandemic and related regulatory adjustments [4]. However, telehealth with video or audio alone does not provide as rich of an interaction as in-person. Adding a social robot to interact with patients during telehealth interactions may be able to improve those interactions. Previous work has shown therapists believe adding a social robot to augment telehealth would significantly improve communication with patients, patient motivation, and patient compliance during telerehab interactions, compared to traditional telepresence [5]. Such a system could be deployed at primary care clinics, schools, or elsewhere in the community to provide access to rehab care for patients who cannot otherwise access it locally. This would enable patients to receive more frequent care without the cost and difficulty of traveling to a distant center of rehab care excellence.

Understanding how patients react to such systems will allow further development and possibly lead to better care for patients. In this paper, we present a case series evaluating the feasibility and acceptability of social robot augmented telepresence (SRAT) for rehabilitation relevant activities, using the social robot, Flo, a small humanoid, with a torso, head, arms, and face. The humanoid rides on a mobile platform with a screen, cameras, and microphones (fig. 1).

Fig. 1.

Flo: Social robot augmenting telepresence for remote rehabilitation.

A. Related Works

This work examines the combination of telerehabilitation and socially assistive robots. Both have rich literature showing the promise of each for delivering rehabilitation care.

1). Social Robots for Upper Extremity Rehab:

Social robots for rehabilitation are not new. They fall within the broader category of socially assistive robots [6] which combine both assistive robots, which support users with disabilities, and social robots, which are designed to interact and communicate with humans. Several systems have been developed for upper extremity rehabilitation. In longitudanal studies, the NT project has demonstrated the viability of SARs for performing autonomous robot guided rehab using a Nao robot to complete pose mirroring and sequence recall games (similar to the Simon says activity used in this study) [7]. The RAC CP Fun project has also demonstrated the viability of similar activities, testing with pediatric subjects both with and without CP [8], [9]. Similarly, but for elder populations, the Bandit robot has been used in multi-session studies to demonstrate that SARs in a rehab context can motivate rehab interactions [10].

2). The importance of presence in interactions:

Robots provide challenges in cost and complexity when being added to a telepresence system. Why then are they worth pursuing? Considerable evidence suggests that physical presence is important for motivation, compliance, understanding, and enjoyment in both rehab and non-rehab interactions. Fridin et al. demonstrated with RAC CP Fun that, comparing an in-person robot and the same robot projected on a screen, pediatric subjects interacted significantly more with the in-person robot [11]. The Bandit robot was similarly used to demonstrate benefits for physical presence in elders [10]. Vasco et al. found that stroke patients prefer using SARs for rehabilitation therapy over virtual agents shown on a screen, reporting higher engagement levels and exercise performance with the physical robot [12]. Mann et al. demonstrated that subjects were more likely to trust, be engaged with, and follow instructions from a robot giving instructions and asking questions, compared to the same interface on a tablet [13]. And Céspedes et al. showed that simply adding a SAR onto a pre-existing neurorehabilitation device as a third agent for motivation and engagement purposes has the potential to increase performance [14]. Together these suggest that adding a social robot as a physically present agent in otherwise virtual (on-screen) interactions, could improve interaction quality.

3). Telerehabilitation:

Telepresence systems can be as simple as using a screen, camera, and Internet connection via a cellphone, tablet, or computer, equipment which patients and providers often already possess. Some systems include a mobile robotic base which can be remotely controlled, such as the systems from Double Robotics and VGo Communications. Others are even more dynamic with robotic appendages to communicate the operator’s intent [15] or screens which can actuate to face the direction the operator is looking [15], [16].

There have been reported successes in telerehabilitation. For example, Dodakian et al. presented a tabletop game system attached to a computer for rehabilitation of stroke patients. By prompting the patient to play physical games while monitoring movements over telepresence, patient compliance and motivation was increased, demonstrating the viability of augmented telerehab [17]. Abel et al. showed that patients’ range of motion can be effectively assessed over telepresence, and that some patients may prefer telehealth appointments for certain rehabilitation tasks, such as range of motion assessments and wound tracking [18]. Bettger et al. in a large longitudanal study (n=287, 12 weeks) found that a tele-physical therapy program for therapy post total knee arthroplasty had lower costs, lower rates of rehospitalization, and was non-inferior in measures of rehab outcomes when compared to a traditional program [19].

However, limitations associated with this technology, including field of view of the operator (clinician), network latency, screen resolution, and projection of three-dimensional interactions into two dimensions, lessen the perception of the presence of the remote operator and reduce spatial reasoning for both users (clinician and patient) [16], [20]. The resulting lack of physical presence, coupled with unclear instructions for movements over telepresence, may decrease patients’ compliance and motivation to perform required motor assessment tasks and, as a result, limit the clinician’s ability to assess the patient’s current function and progress and motivate home therapy adherence. This highlights a need to develop platforms that have a physical presence and can perform both assistive and social functions. With the current pandemic, the need and call for telerehab systems has grown [21], a need which will continue to grow with shifting demographics and future unforeseen care challenges.

B. Social Robot Augmented Telepresence

Recognizing the broad potential impact from telerehab and that some of the challenges with telepresence could be overcome by using a socially assistive robot, we propose to augment telerehab interactions with social robots. In a previous study of 351 therapists in the United States, therapists reported they believe social robot augmented telepresence (SRAT) would significantly improve communication, motivation, and compliance during telerehab interactions, compared to classical telepresence. To explore this new direction in healthcare robotics, we have developed Flo, an example SRAT.

The system is comprised of a Kobuki drive base, two RealSense D415 cameras, a screen, speakers, and a fisheye camera (fig. 1). A humanoid with a body, head, arms, and a face can be placed onto the system and easily removed, allowing the system to be used in both classical telepresence (CT) and SRAT configurations. The system is controlled via a remote web interface by a plays and script methodology with wizard of oz control. The humanoid can synthesize arbitrary speech and the face of the humanoid can be dynamically changed [22]. More design details can be found in [23].

Here we explore the feasibility of social robot augmented telepresence for engaging in upper extremity rehabilitation activities, using the Flo platform to compare SRAT to CT.

II. PROCEDURE

The study was approved by the University of Pennsylvania Institutional Review Board. In addition to consenting to participate in the study, all subjects also provided an optional media release, allowing publication of their images. A convenience sample of subjects, over the age of 4, either with or without upper extremity impairment, was recruited. Broad enrollment criteria were used to search for initial evidence for feasibility, or lack thereof, among the populations of interest.

A. Demographics and Baselines

After consenting, subjects were assessed using the box and block test [24] to measure unilateral gross manual dexterity, children’s color trails test for pediatric subjects [25] or color trails test for adult subjects [26] to measure visual attention, graphomotor sequencing, psychomotor speed, and cognitive flexibility, as a proxy for executive cognitive function more generally, and grip strength test [27] to measure hand and forearm strength as a proxy for upper limb strength (fig. 2).

Fig. 2.

Subjects performing the clinical assessments at the beginning of the trial. From left to right: the box and block test, color trails test, and grip strength test.

Subjects, aided by a caretaker/parent if present, answered a survey asking: basic demographics, history of cognitive and motor impairment, technology usage, level of education, and current therapy practice and compliance, how they are feeling today, and how they feel about robots. Surveys were administered by a study team member, enabling data to be collected from subjects of all ages and cognitive levels as well as allowing follow-up questions to be asked.

B. Conditions and Ordering

Subjects were randomly assigned to an ordering group. With FTF signifying face-to-face (fig. 3), SRAT (fig. 4), and CT (fig. 5), the groups were: FTF-SRAT-CT and FTF-CT-SRAT. Once the intake surveys were complete, the subject was seated in a room. For the FTF condition, the operator entered the room and sat on a chair, next to the telepresence system, used to gather video data, in front of the subject. For the SRAT/CT conditions, the robot entered the room under remote control by the same operator. This ordering sequence was used to give each subject an initial baseline face-to-face experience to begin, as would likely happen in the real world. Previous results have suggested that when initial interactions are not in-person, results are poor [11].

Fig. 3.

Subject interacting in person with the operator (Condition FTF). On the left, playing a Simon Says game. On the right, performing the target touch activity.

Fig. 4.

Subject interacting with the operator via telepresence with a social robot augmenting the interaction (Condition SRAT).

Fig. 5.

Subject interacting with the operator via classic telepresence (Condition CT).

C. Interaction

To begin each interaction, the operator introduced themselves and, in the SRAT condition, the robot. The operator asked the subject if they wanted to play a game. When the subject said yes, in the SRAT condition the robot said “great, let’s play Simon Says … In Simon says, I will tell you something to do and show you how to do it, mirrored. If I say Simon says, you should do it with me. If I do not say Simon says, you should not do the action. Watch out, I may try to trick you. After every movement, return to a ready position.”. In the CT/FTF conditions, the operator provided the instructions. The game was played with the subject moving through 20 motions with an additional 6 repeated motions where there was no Simon Says command, with all instructions given by the robot in the SRAT condition. The motions were identical in all conditions, composed of bimanual activities: clapping, reaching overhead with hands apart, reaching out forward with arms, covering both eyes and unimanual activities: reaching across the body to touch the shoulder, touching the mouth, touching the top of the head, reaching out to the side, etc. This game pushed the subjects to the ends of their range of motion, using motions which are relevant for activities of daily living (ADLs). For the first four subjects, the unimanual activities were done individually for each arm, randomly ordered. As the study evolved, for the final two subjects, activities were made more difficult and engaging using bilateral non-symmetrical motions constructed randomly (ex: “wave with your right arm and touch your shoulder with your left hand”). This change was made in response to perceived boredom among subjects and subjects’ apparent ability to memorize motions between interactions, no longer requiring visual information to complete the activity.

The operator then said that they want the subject to play another game. In the CT/FTF conditions, the operator explained the game and in the SRAT condition, the robot explained it: “In the target touch activity I will tell you to touch the dots on my [hands/board] … Let’s start in a ready position, return to this position after every touch”. In the CT/FTF condition, the telepresence system had colored dots mounted on a board hanging on the robot. In the SRAT condition, the robot had the same dots on its hands and moved its hands to the same points as those on the board. This activity tested the attention and motivation of the subjects by instructing them to do repetitive motions. The first four subjects were instructed to touch each of three dots with each hand ten times, for a total of sixty point-to-point motions. The final two subjects were instructed to complete a series of 1 to 4 dot/hand sequences with two dots available, randomly selected. In the modification made for the final two subjects, the activity proceeded more quickly, motion was more diverse (the same dot was not touched repeatedly), difficulty was increased, and more sustained attention was required.

These activities were used to evaluate two different areas of motor function, range of motion and ballistic movement. Activities like the Simon says game appear repeatedly in the SAR literature for use with both children and elders.

D. Mid-Interaction Survey and Rest

On completion of each condition, the robot/operator exited the room. Subjects were then given a survey asking questions from the NASA task load index (TLX) [28], intrinsic motivation inventory (IMI) [29], how well they enjoyed the interaction, if they would like to do the interaction again, and how safe they felt during the interaction. Once the surveys were completed, subjects rested until at least 15 minutes had passed since the end of the prior condition.

E. Post Study Survey

At the end of the study, subjects are asked questions about the entire experiment: which interaction modality they thought was best and several questions adapted from Telemedicine Satisfaction and Usefulness Questionnaire [30] on whether they thought telemedicine would change how they manage their healthcare, communicate with their clinicians, and if telehealth visits would be convenient for them.

III. RESULTS

A. Subjects

Eight subjects participated in the pilot trial. The surveys were dramatically edited between the first and second subjects, so the first subject was excluded from analysis. For one other subject, the power system on the robot failed, preventing that subject from completing the protocol, so they were excluded. Therefore 6 subjects’ results were analyzed. Complete subject information can be seen in table I.

TABLE I.

Subject demographics, affective state, experience/feelings regarding relevant technologies, and trial interaction preferences

	AJ	HL	GS	VM	PD	BF
Diagnosis	Stroke	Stroke	Stroke	None	None	None
Age	55	53	63	13	4	6
Gender	F	M	F	M	F	F
Color Trails Test Z-score
CTT-1	−3.9	−2.2	0.0	−0.8	-	-
CTT-2	−5.0	−1.5	0.9	0.4	-	-
Box and Block Test Z-score
Dominant hand	−4.6	−3.1	−5.6	−3.2	0.2	−3.2
Non-dominant hand	−2.3	−2.3	−5.2	−3.5	0.3	−3.1
Self-Assessment Manikin
Valence (1: Happy, 9: Unhappy)	1	1	1	3	1	1
Arousal (1: Excited, 9: Relaxed/Sleepy)	1	1	5	4	1	3
Dominance (1: Submissive, 9: Dominant)	4	7	9	7	9	9
Please rate your level of experience with (1: No experience, 5: Very high experience):
Computers	4	2	3	4	2	2
Tablets	4	2	3	4	3	3
Smartphones	4	2	2	4	3	1
Robots	4	2	1	3	2	1
Have you ever done:
A video call?	Yes	Yes	Yes	Yes	Yes	Yes
A video call for healthcare?	No	No	Yes	No	No	No
How do you feel about (1: Very negative, 5: Very positive):
Using video calls for healthcare?	5	4	2	4	5	2
Robots?	5	5	4	5	5	4
Interaction Modality Preference:
First	CT	FTF	FTF	SRAT	SRAT	SRAT
Second	SRAT	SRAT	SRAT	FTF	CT	FTF

The subjects with stroke varied functionally – one had aphasia, another had spasticity and loss of function in one hand, preventing independent finger and wrist actuation. All but one subject (PD) performed 2 to 6 standard deviations below normal on the box and block test [24], [31]. Subjects varied from severely impaired to typically functioning on the color trails test. At intake, all subjects reported being happy, excited to neutral, and dominant to neutral. All subjects reported positive feelings towards robots. All subjects had previously made video calls, only one had done so for healthcare (GS). Subjects initially were mixed on their feelings towards using video calls for healthcare. AJ, BF completed the study in the FTF-CT-SRAT order. HL, GS, VM, PD completed it in the FTF-SRAT-CT order.

B. Ability to Participate

All subjects were able to fully participate with the robot. No subjects reported difficulty understanding the instructions given by the humanoid. AJ reported difficulty in the CT condition with seeing the operator due to the small screen size. The last two subjects to participate (PD and BF), who were the youngest and experienced more challenging activities, appeared to have more difficulty with the Simon Says activity and had significant difficulty with the target touch activity across conditions. PD was not comfortable with left and right hands and had difficulty with more than two step instructions.

C. Quality of Interactions

Using the IMI (1 low to 7 high), enjoyment for each condition averaged between 5.1 and 5.8, competence averaged between 6.5 and 6.7, effort averaged between 3.8 and 4.4, and pressure averaged between 2.1 and 2.4.

D. Modality Preference

All three pediatric subjects rated SRAT as the best modality. The three adult subjects rated SRAT as the second-best modality. Complete ratings are shown in table I. AJ rated CT as their preferred modality, reporting that it was the lowest quality interaction but sufficient to complete the activities and likely the lowest cost, so best overall.

E. Observations

Throughout the interactions, the subjects had opportunities to provide feedback and the study staff recorded notes. GS reported not liking telepresence, but liked working with the social robot, and stated that they felt less pressure as the robot could adjust its pacing to their needs (compared to a human who may be in a rush). AJ said that they thought the social robot would be useful for home practice. VM reported that the SRAT interaction was the best due to it being the funniest. BF was more energetic filling out surveys after the SRAT interaction than the other interactions. PD, the youngest subject, was very excited to interact with the social robot. After completing the experiment and surveys, PD asked if they could play with the robot more. We drove it into the room and had the robot say hello, wave, and talk with PD. PD interacted with it and ended up petting it on its head as if it were a pet or a doll.

IV. DISCUSSION

This case series included users of a wide variety of ages and motor and cognitive function. All subjects were able to participate fully in three conditions: face-to-face, classical telepresence, and social robot augmented telepresence. Enjoyment was high for all conditions. All conditions required moderate effort, but subjects reported high competence and felt low pressure. With this size of a case series, comparisons between measures after conditions are not warranted. The high level of enjoyment and the ability of all subjects to interact with the social robot while it augmented the telepresence interaction demonstrates acceptability of the system by patients. All trials were completed in a laboratory setting within a hospital, which is similar to an exercise room in an elder care facility or school, and so approximates the type of space where these technologies might be used.

The social robot augmented telepresence system was rated higher than traditional telepresence by five of the six subjects. This is an early indicator of the benefit that a social robot can add. The pediatric subjects all rated the social robot augmented condition as best. This may be partly a novelty effect, which is well documented in the literature. But previous systems, such as the NT, have demonstrated the use of social robots for rehab longitudinally with good results [7]. Even if there is a large amount of novelty driven interest, by using evolving games and activities coupled with dynamic robot expressions, high engagement should be maintained over time. Seeing the youngest subject’s (PD) high enthusiasm for the social robot was exciting to the study team. That subject clearly assigned a personality to the robot. One of the adult subjects commented on the social robot being well suited for subjects who move slower, because it can adapt to their pace, reducing the pressure during visits while encouraging improved performance. This highlighted a point of utility that we had not anticipated and adds more rationale for developing social robots for rehabilitation and remembering to design with elders, not just children, in mind.

A. Limitations

All subjects had positive views towards robots at the start of the trial and had previous experience making video calls, which may have made them more likely to accept SRAT. Because this trial was conducted during the COVID-19 pandemic, all participants wore masks during interactions, likely decreasin the quality of face-to-face interactions. This is representative of how real rehab was conducted for at least two years. It is possible that if the interactions had been more challenging and engaging for the first four subjects, then they would have received SRAT even better.

V. CONCLUSIONS

We completed a case series with 6 subjects interacting with the Flo robotic system, an example of social robot augmented telepresence. By multiple metrics, the system performed well, was accepted by subjects, and appeared feasible to use for upper extremity motor rehabilitation activities and assessments.