Design and Evaluation of an Interactive Exercise Coaching System for Older Adults: Lessons Learned

Abstract

Although the positive effects of exercise on the well-being and quality of independent living for older adults are well-accepted, many elderly individuals lack access to exercise facilities, or the skills and motivation to perform exercise at home. To provide a more engaging environment that promotes physical activity, various fitness applications have been proposed. Many of the available products, however, are geared toward a younger population and are not appropriate or engaging for an older population. To address these issues, we developed an automated interactive exercise coaching system using the Microsoft Kinect. The coaching system guides users through a series of video exercises, tracks and measures their movements, provides real-time feedback, and records their performance over time. Our system consists of exercises to improve balance, flexibility, strength, and endurance, with the aim of reducing fall risk and improving performance of daily activities. In this paper, we report on the development of the exercise system, discuss the results of our recent field pilot study with six independently-living elderly individuals, and highlight the lessons learned relating to the in-home system setup, user tracking, feedback, and exercise performance evaluation.

I. Introduction

Providing care for populations at risk such as older adults and those with neurodegenerative diseases is a major societal challenge. This challenge can be partially mitigated by improving the health-related behaviors of older adults. In addition to nutrition and socialization, physical and cognitive exercises are among the most effective preventive approaches that improve outcomes as well as quality of life [1]. Despite these benefits, many older adults continue to face difficulty in engaging in exercise activity due to various social and economic reasons, including motivation, confidence and skills. One approach that can be used to address these barriers is using coaching in conjunction with a variety of health behavior change techniques, including motivational interviewing [2]. Unfortunately, a personal coach is typically not economically feasible, and access to exercise classes in community centers is not an option for many older adults.

One promising approach that mitigates these problems is the use of a semi-automated coaching [3], [4] in conjunction with automatic exercise monitoring and health-coaching system. In our prior work, we developed a prototype of such a semiautomated coaching system that comprises unobtrusive sensing of each participants’ behaviors and combines it with artificial intelligence tools aiding the coach in crafting individualized messages to the participants. This original coaching system was focused on cognitive and physical exercises, sleep, and socialization. Although the participants were encouraged to exercise using YouTube videos, this system lacked the ability to track and provide real-time feedback for physical exercises. This functionality is important in replicating the benefits of a personal exercise coach.

Fortunately, with the introduction of more affordable motion sensing technology and better video graphics it is now possible to develop home-based systems that enable coaching of both physical and cognitive exercises. Such a possibility is collaborated by the explosion of exergames that combine exercises with video games to enhance motivation and promote more active lifestyle for wide population segments [5], [6]. Several commercial fitness applications have been released to date, such as Your Shape: Fitness Evolved (Ubisoft Montreal, 2010), UFC Personnal Trainer (Heavy Iron Studios, 2011), Nike+ Kinect Training (Microsoft Studios, 2012) etc. Unfortunately, many of the available commercial fitness applications are geared toward a younger population and are too difficult and not sufficiently engaging for older users. In addition, older adults often require specially designed user interfaces due to inherent sensory and cognitive difficulties [7], [8].

Several research studies pertaining to well-being and exercise in older adults applied off-the-shelf technologies or developed customized systems to study short- and long-term effects of computer-assisted exercise. Doyle et al. [9] introduced a proof-of-concept custom-designed interactive vision-based system that delivered balance and strength exercises to older adults. Even though the proposed technology was aimed for elderly to use at home, the authors report only the results from in-lab testing with a small number of participants. Tsai et al. [10] recently assessed the acceptability of a fitness testing platform, iFit, for installation in an assisted living community with the aim of promoting fitness and slowing the onset of frailty. Macek et al. [11] proposed a custom-designed physical exercise system using an ultrasound sensor and heart-rate monitoring device. The authors stressed the importance of collecting continuous physical measures for performance analysis during the exercise. Doyle et al. [12] presented a smartphone-based exercise system where they emphasized the importance of visual feedback on quantitative measures in exergaming.

In recent work Garcia et al. [13] and Pisan et al. [14] used Microsoft Kinect camera in stepping-exercise system targeting older population to predict the loss of balance under cognitive load. The physical health and cognitive abilities of the participants were assessed using in-exercise collected data and standardized clinical measures. The qualitative evaluation of the system, however, was reported only from the initial laboratory testing. Similar proof-of-concept systems based on the Microsoft Kinect camera were also presented by Lange et al. [15] and Ganesan et al. [16], who both reported only on usability and data gathered in a laboratory setting. For a more comprehensive survey of interactive exercise research in older adults, we refer the reader to several systematic reviews found in literature [17]–[19].

In summary, although several researchers have studied interactive exercise in older adults, the majority used commercial games without any direct measurements of the physical activity or the participants were tested only in a controlled laboratory environment. This is unfortunate because in order to capture the time-varying characteristics of the exercise performance and to better understand the changes over time, continuous physical measurements (e.g., joint angles, positions, velocities) are needed. Furthermore, the higher level statistics of these measurements (e.g., average, standard deviation, variability), also referred to as internal outcome measures [20], can provide additional parameters for quantitative analysis.

The algorithms that capture and quantify exercise activity in commercial products are usually proprietary and are typically inaccessible for post-hoc analysis. As a result, most of the existing research studies rely on clinical assessment based on external outcome measures that include various standardized scales and subjective questionnaires to quantify the effect of such intervention. These measures, obtained only intermittently, do not provide any direct information on physical performance during the exercise, nor can they be used for just-in-time coaching. To alleviate this issue, researchers used additional sensors (e.g., body-worn motion trackers) to measure the activity during commercial game play [21], subject to the usual drawbacks, including synchronization and usability issues.

The main focus of this paper is therefore on the research necessary to design and deploy a general architecture for a computer-based interactive exercise system with real-time assessment and guidance in combination with remote coaching approaches [3], [4]. This work extends the initial presentation of our exercise system in [22] by including additional details on the design, implementation and deployment. Our interactive exercise system is based on the Microsoft Kinect camera that records user's entire body kinematics and provides automated visual and auditory feedback on the performance. To assess the acceptability and usability and to examine the feasibility outside laboratory environment, we deployed the system into six homes of elderly individuals who used it during a time period of two to six weeks. Based on this experience, we provide guidelines for future development of interactive exercise systems for older users with vision-based motion-tracking technology. The particular lessons learned and reported in this paper include (1) limitations on living space with respect to deployed technology, (2) usability issues with the camera and software, (3) usability issues with the user interfaces, (4) the form of visual and auditory feedback, and (5) interpretation of physical measurements from the captured kinematics in relation to clinical outcome measures and standard components of physical fitness, such as balance, flexibility, strength, and endurance [23].

The remainder of this paper is organized as follows. Section II describes the design of the proposed automated coaching system based on the Kinect camera. The pilot study design is presented in Section III, and the results of the pilot study are evaluated in Section IV. User feedback and lessons learned are elaborated in Section V, and finally, future work is discussed in Section VI.

II. Exercise System Design

Our goal in this research was to develop an architecture for automated exercise coaching of older users that would motivate them and track their physical performance while being able to exercise in their home. The initial phase included detailed needs analysis for the complete coaching system that included focus groups with older adults, caregivers, coaches, etc. [3]. In this paper, however, we focus on the resulting requirements restricted to the interactive exercise coaching system as follows:

• Use of unobtrusive, low-maintenance, and low-cost measurement modality;

• Inclusion of age-appropriate exercises for older adults;

• Development of an elderly-friendly user interface;

• The ability to obtain raw measurements that would provide estimations of balance, endurance, strength, and flexibility [23];

• Compatibility for future integration with the coaching platform.

Although we considered other options including wearable devices, to meet the requirements for an unobtrusive, low-cost and low-maintenance solution, we opted for the Microsoft Kinect camera [24]. This approach allowed us to develop a customized solution, as opposed to using a commercially available fitness application.

A. Usability, User Interface and Control Flow

The first implementation was based on a PC with an LCD screen controlled by a variety of possible navigation methods. In adherence to the design recommendation for older users, the key design objective was set to accommodate potentially reduced cognitive, sensory, and motor capabilities [7]. Accordingly, the amount of information and choices presented on each screen were minimized. The color and size of text was carefully selected to provide high contrast and visibility since the users would operate the system from a distance. Switching between screens was controlled by a set of buttons that were consistently positioned at the bottom of the display. To address the possibility of different sensory impairments, we implemented various modes of navigation, such as keyboard, mouse, remote control, and speech. All of these modalities were tested at various stages of the development and deployment. Gesture controls were avoided because of the potential difficulties due to various motor impairments that affect elderly. The user interface was implemented in C++ programming language using OGRE 3D graphics library [25].

Figure 1 shows the control flow of the user interface. When the exercise software is started, the user is presented with the main screen with a personalized greeting from where he/she can watch several introductory videos, start the warm-up, or start a new exercise session. The screen also provides access to the camera setup where the Kinect camera tilt can be adjusted. The exercise session consists of a sequence of 12 exercises (more details on the exercise selection are provided in Section III-B). Each exercise consists of three stages: (1) instructions - including a demonstration of the routine, (2) execution - computer-assisted exercise by the user, and (3) exercise summary - the overall evaluation of the performance.

Fig. 1.

User interface control flow displaying the main screens of the exercise system. The bottom three screens were common for each of the 12 exercises in the program.

Each exercise is presented in a form of video instructions of the coach explaining the health benefits of the exercise and demonstrating the routine. The users can at any time start the exercise once they familiarize themselves with the instructions. After watching the instructions, users perform the exercise on their own by following the video feedback. The performance is measured in real-time using the Kinect skeleton representation while various corrective alerts are provided by the system via textual and computer-generated audio cues. More details on the movement analysis and feedback are provided in Sections II-B and II-C. Once the user completes a predefined number of repetitions or selects the “End Exercise” button, the display of that exercise ends, and the performance is recorded into the database. Afterwards, a message of encouragement is briefly displayed showing the remaining number of exercises followed by the summary screen. The summary screen presents a weekly chart with performance scores observed in the particular exercise. The user can also view monthly or the last 10-session summary.

B. Real-time Movement Analysis

We used the skeleton representation from the Microsoft Kinect to measure the movement kinematics. The Kinect is a depth-sensing camera that provides relatively robust 3D reconstruction even in low-light conditions [24]. The accompanying Kinect software development kit (Kinect SDK) provides segmentation of human blobs even in cluttered scenes and real-time pose estimation with the 3D locations of 20 joints [26]. In our architecture the Kinect camera pose estimation is applied in three areas: (1) to quantify the performance of each exercise, (2) to support repetition counting, and (3) to trigger feedback alerts that attempt to correct an exercise performance. The processing of the raw skeletal data is performed in real time as shown in Figure 2.

Fig. 2.

Overview of the data processing pipeline for automated coaching. Kinect 3D joint position data are processed into higher-level performance measures which are in turn used to verify exercise execution, count repetitions, and provide feedback on performance.

To quantify the performance of each exercise, we first extracted measurement primitives from the skeletal data, i.e., joint angles, angles relative to the vertical or horizontal plane, distances, absolute positions, etc. These measurement primitives for a particular exercise were chosen based on the goals of the specific exercise (i.e. primary measurements). These internal, hidden variables were also used to count repetitions, trigger alert messages, compute movement features and generate overt feedback to the user. The actual performance metrics were derived by potentially non-linear combinations of several input primitives. To represent the participants’ relative performance, the primary measurements were normalized to form movement scores bounded by the range of 0 and 1, based on the maximum range of the measurement primitives (e.g., range of motion of elbow flexion between 0 and 90 degrees translates into a score of 0 to 1). Since the Kinect output is noisy, the extracted exercise-specific metrics were smoothed by moving average filters.

To track execution of exercise repetitions, it was necessary to divide each exercise into stages identified by the exercise stage detection module that was designed to uniquely determine the instantiated exercise stage from the pose measurements. In this model, the stage transitions can be triggered by a single or combination of several different measurements or scores, detected as transitions either using a fixed threshold or a dynamically adjusted threshold controlled by a hysteresis function. Again, to improve the robustness of this approach the stages were filtered using a time constant to define the expected minimal duration of each stage. In principle, these inferences could be learned using standard machine learning techniques, but the small amounts of data and high variability would present considerable challenges to the machine learning approaches.

Finally, the correctness of exercise execution was checked based on selected complex pose measurements. In each stage, we defined constraints of the movement, such as keeping the hands close to each other, keeping the feet on the floor, maintaining the torso in upright position, etc. The constraints were defined as thresholds for selected pose measurements which can be based on a single measurement primitive or a combination of multiples as mentioned above. If the constraint was violated, the alert module executed a message via textual and audio cues as described in the next section.

As an example, Figure 3 shows the Buddha's Prayer exercise [27] which consists of lifting the arms upward and downward while keeping the palms close together and sitting tall in the chair. The goal of the exercise is to lift the hands as high as possible without loosing contact between the palms. Figure 3a shows the relevant joints involved in the exercise. The primary measurement M is defined as the minimum vertical distance of the left (LW(z)) and right wrist (RW(z)) as measured from the ground plane. The minimum is chosen in order to ensure that sufficient lift is exerted in both hands. The corresponding score, which is shown on the screen via the performance bar (Figure 4), is calculated as a ratio of the difference of vertical distance between shoulders and current wrist position, normalized by the length of the arm, which corresponds to the maximal possible lift.

Fig. 3.

Exercise analysis example for the exercise Buddha's Prayer: (a) performance evaluation is done by extracting primary measurements related to the exercise; (b) exercise execution is verified using hidden measurements and corrective feedback is provided to the user; (c) stage detection determines the stage of the exercise and tracks number of repetitions.

Fig. 4.

Feedback screen displayed during the exercise performance includes video of the instructor, video captured by the Kinect camera, and various feedback elements related to the exercise performance on the top-left side of the screen.

Figure 3b shows the verification of the exercise correctness (keep the palms close) which is determined from the hidden measurement. The hidden measurement in this case is defined as the Euclidean distance d between the two wrist positions. If the distance exceeds the threshold, the messaging system will display the alert. In addition, the exercise requires subject's upright sitting posture, therefore, the system also tracks the tilt angles with respect to the ground normal for the upper body.

Finally, Figure 3c shows the stage detection which is used to track exercise repetitions. The stage detection alternates between two stages (1 and 2) through a hysteresis function. If the current exercise stage is set to 1 and the primary measurement is increasing, the stage detector will transition to stage 2 once the hysteresis threshold Δ is exceeded. Once the stage detector is in stage 2, the transition to stage 1 will occur only if the primary measurement is decreasing and exceeds its value reduced for the hysteresis threshold. Table I shows the measurements collected for each exercise.

TABLE I.

Overview of physical exercises included in the interactive coaching system [27]

Ex	Exercise Name	Movement	Body Part	Benefit	Goal	Internal Measures	Related External Measures
1	Heel Drags, Left Leg	Drag heel along floor, toward chair	Knee flexion	Strengthens knees, enhances gait, makes getting out of a chair easier	Strength	Forward leg reach, Backward leg reach	SFT: chair stand; BBS: sitting to standing, standing to sitting, sitting unsupported, standing unsupported, retrieving object from floor
2	Heel Drags, Right Leg
3	Lateral Steps, Left Leg	Step out to side	Hip abduction	Reduces lateral fall risk	Strength, Flexibility	Upper leg rotation	BBS: turning to look behind, standing on one foot, turning 360 degrees, standing with one foot in front, placing alternate foot on stool
4	Lateral Steps, Right Leg
5	Cross-Legged Extensions	Lift legs while crossed	Knee extension	Strengthens knees, enhances gait, makes getting out of a chair easier	Strength	Heel elevation	SFT: chair stand; BBS: sitting to standing, standing to sitting, sitting unsupported, standing unsupported, retrieving object from floor
6	Abs-in Knee Lift	Alternate knee lifts, with tall back	Hip flexion	Improves balance, posture, and core stability in gait	Balance, Strength	Lower leg lift	SFT: 2-min step in place, 4-foot up and go; BBS: placing alternate foot on stool
7	Cops and Robbers	Arms back, forearms up, arms out at shoulder level	Scapular depression, retraction	Improves shoulder flexibility, helps with driving	Flexibility	Shoulder rotation, Arm forward reach	SFT: back stretch; BBS: sitting unsupported
8	Buddha's Prayer	Hands together, arms lift up and down	Mid back and core	Flattens thoracic curve, improves posture for balance, reaching	Strength, Flexibility	Wrist elevation	SFT: arm curl, back stretch; BBS: reaching forward with outstretched arm, sitting unsupported
9	Teeter-Totter Chair Stands	Rise from a chair and sit back down	Full body	Improves functional ADL skills: hip extension, balance, transition	Strength, Balance	Shoulder elevation	SFT: chair stand, 2-min step in place, 4-foot up and go; BBS: sitting to standing, standing to sitting, standing unsupported, transfers, retrieving object from floor
10	Shallow Squats	Hinge at hip, pushing hips back over chair as prop	Hips and knees	Helps getting up from floor, getting in and out of cars, chairs	Strength, Balance	Squat depth	SFT: chair stand, 4-foot up and go; BBS: sitting to standing, standing unsupported, standing to sitting, transfers, retrieving object from floor
11	Heel Raises (standing)	Lift heels off ground	Calves and ankles	Enhances gait, fall prevention, builds core strength	Strength, Balance	Shoulder elevation	SFT: chair stand, 2-min step in place; BBS: sitting to standing, standing unsupported
12	Marching in Place	Step in place	Full body	Improves step length and height, highly functional for ADL	Strength, Balance	Lower leg lift	SFT: chair stand, 2-min step in place, 4-foot up and go; BBS: transfers, standing unsupported, standing on one foot, standing with feet together, placing alternate foot on stool, standing with one foot in front

C. In-exercise Feedback

During the design phase we explored several visualization options on how to present the instructions from the coach and in-exercise feedback to the users. We considered (1) full-screen video of the coach with Kinect video insert, (2) video of the coach with Kinect 3D data presented side by side, and (3) Kinect video and skeletal data with pictorial cues of the exercise. Informal usability assessment with coaches and several users suggested a preference for a full-screen video visualization.

For the in-exercise feedback, the video is overlaid with additional feedback data as shown in Figure 4. In the top portion of the screen, the consecutive number of the exercise, the exercise name, and current repetition count are presented. In the lower-right corner, a video of the user captured by the Kinect camera is displayed for reference. To provide assistive feedback during the exercise, the user is also presented with visual cues on the targeted performance measurements which are displayed in a form of a dynamic performance bar whose length corresponds to the performance score as defined by the given complex pose measurement. A white vertical line denotes the maximum value achieved during a particular session. Below the performance bars, we display the corrective and encouraging message cues. The messages are also conveyed to the user via text-to-speech synthesis. We use additional audio cues (short “ding” sounds) whenever movement repetition is successfully counted.

D. Database

The data storage for the exercise system was implemented locally using MySQL [28]. For each user the system tracked the start and end time of each exercise session. Within the session, data in each time instance (at approx. 30 Hz) consisted of raw Kinect skeletal data for 3D positions of 20 joints, current value of observed primary and hidden measurements, position of the video feedback, current exercise stage, current repetition count, and feedback messages if provided. Overall exercise performance was stored in the summary table which included name of the exercise, number of completed repetitions, and observed primary measurements with corresponding name, minimal and maximal values.

III. Pilot Study Design

A. Study Participants

The study participants were selected from a pool of subjects of the Oregon Center for Aging & Technology (ORCATECH) Living Lab at Oregon Health & Science University [29] who were also enrolled in a health-coaching program. A total of six independently living elderly subjects (2 male, 4 female) were included in the pilot study. All the subjects were Caucasian with a mean age 81, ranging from 74 to 91. The selection criteria included the ability to (1) speak and read English, (2) understand and follow instructions, and (3) ambulate without any assistive devices. All the participants had normal or corrected vision and none of the participants had any known health problems that would prevent them from using the designed system safely on their own.

B. Exercise Program

We based our exercise program on Sue Scott's Able Bodies Balance Training book [27] which offers more than 130 exercises to enhance balance, flexibility, strength, and endurance in older adults. We selected 12 exercises that are aimed to reduce fall risk, increase balance confidence, and have positive impact on activities of daily living (ADL). Table I provides the list of selected exercises with detailed information on each exercise. The exercise selection was also determined by considering robust tracking with the Kinect camera, based on our initial tests [30], while providing sufficient variety of exercises that engage different parts of the body.

C. Performance Assessment - Outcome Measures

During the course of the pilot study the participants’ exercise performance was continuously monitored by the coaching system using the primary measurements extracted from the Kinect pose estimation as defined in Section II-B. We converted these primary measurements into several internal outcome measures by analyzing higher-level statistics, such as averages, deviation and variability to assess and provide feedback about the participants’ overall performance of each exercise in each session as well as across sessions.

We also evaluated the participants’ physical abilities at the beginning and at the end of the coaching intervention using several external outcome measures which included standardized clinical tests, such as the Senior Fitness Test (SFT) developed by Rikli and Jones [31] and the Berg Balance Scale (BBS) [32]. The Senior Fitness Test (SFT) measures the physical parameters associated with functional ability and identifies the risk for loss of the ability to perform specific ADL. We included the following four items in our pilot study: (1) the chair-stand test (CST) and (2) arm-curl test (ACT) for measurement of muscle strength in lower and upper body, respectively; (3) the 4-foot up-and-go test¹ (4FUGT) for assessment of agility and balance; and (4) the 2-minute step-in-place test (2MSPT) for evaluation of aerobic endurance. The Berg Balance Scale (BBS) on the other hand is a 14-item scale developed for measuring both static and dynamic balance abilities of older adults. Table I provides a list of internal and external outcome measures for each exercise.

D. Experimental Protocol

A phone interview was first conducted to determine which participants were appropriate for the pilot study based on health-related inclusion criteria and availability of physical space to accommodate the Kinect camera system. Selected participants were initially visited by the health coach to obtain informed consent and further information on general health, physical activity levels, and computer usage. Next, the health coach administered the initial fitness tests (i.e., SFT and BBS) and recorded participant's pre-intervention scores. The exercise system with the Kinect camera was then installed and the participant was given training on how to use the system. The health coach led the participant through the exercises and provided specific instructions to augment provided instructional videos. The participants were instructed to complete at least 10 sessions and were suggested to exercise preferably once every other day during the course of the enrollment. During this time, the health coach checked in with the participants on weekly basis to note any issues with the exercise intervention or the system. Afterwards, post-intervention tests and usability feedback was gathered and the system was removed.

IV. Pilot Study Results

A. Data Collection Overview

Table II provides a summary of the user experience with the exercise system. We can observe that the duration of the exercise regimen varied considerably between the participants. Some participants (e.g., participants #3 and #5) had the exercise system in their homes for a longer term (approx. 6 weeks) whereas others (e.g., participants #4 and #6) completed their sessions in a shorter term (approx. 2 weeks). We can see from ESU rates that some participants (e.g., participants #4 and #6) used the system more frequently than others, even exceeded the recommended exercise regimen. On the other hand, the analysis of EER rates and completions of individual exercises (not shown in the paper) reflect that some participants (e.g., participants #5 and #6) struggled to complete the required number of repetitions in several exercises or concluded their workout sessions by skipping a few of the exercises entirely. Since one of our goals was to obtain information on system usage without providing restrictions on the exercise regimen, the resultant exercise data are not fully controlled and thus the participants cannot be easily compared. Due to limited space, we illustrate several different analysis directions and present a sample of results based on the pilot study data.

TABLE II.

Overall User Experience With The System

		D	S / Ŝ	SCE/SC̃E/SĈE	EER	ESU
Participants	1	29	12 / 15	118 / 144 / 180	82%	66%
	2	28	12 / 14	121 / 144 / 168	84%	72%
	3	44	11 / 22	130 / 132 / 264	98%	49%
	4	17	10 / 9	112 / 120 / 108	93%	104%
	5	44	16 / 22	124 / 192 / 264	65%	47%
	6	14	11 / 7	99 / 132 / 84	75%	1 18%

D: # days the system was deployed in a home.

S: # sessions (days) the system was used for exercise by a user.

Ŝ: [D/2], expected # sessions under exercise-every-other-day regimen.

SCE: total # successfully completed exercises by a user in S sessions.

SC̃E: 12 × S, expected SCE by a user in S sessions.

SĈE: 12 × S, expected SCE by a user in Ŝ sessions.

EER: SCE/SC̃E, effective exercise rate.

ESU: SCE/SĈE, effective system usage.

B. Evaluation of Internal Outcome Measures

In this section, we provide an example of an internal outcome measure which evaluates the in-exercise performance from the Kinect measurements for one subject, as described in Section II-B. Figure 5 shows the extracted primary measurement of wrist lift for the Buddha's Prayer exercise as well as the corresponding detection of stages, repetition counts, and selected skeletal key frames, as performed by participant #6 on day 11.

Fig. 5.

Primary measurement example for Buddha's Prayer (participant #6; day 11): (a) wrist lift (dashed curve), stage detection (dark/light shades), repetition counts (solid vertical lines), and maximum/minimum values of the wrist lift measurement in each exercise repetition (diamonds/squares, respectively), (b)-(g) skeletal poses for key frames marked with cross markers in (a).

Figure 6 presents a longitudinal, exercise-specific evaluation of the participant's performance across all exercise sessions. In this particular case, we review results of the Buddha's Prayer exercise performed by participant #6. The analysis of repetition times can provide information on general flexibility, strength, and endurance levels both during an exercise session and across multiple sessions. For instance, Figure 6a shows a slight decrease in the wrist lift measurements across multiple exercise sessions. This trend roughly corresponds to the reduction of the average exercise repetition times across multiple sessions as shown in Figure 6b. Since this observation provides only a coarse correlation with individual's flexibility and strength levels, it is difficult to directly infer overall trends in flexibility and strength changes.

Fig. 6.

Analysis of the wrist lift measurements across all exercise sessions as observed in participant #6: (a) box plot of the maximum distance traveled by the wrists in each repetition, (b) box plot of the time taken to complete each repetition. Circular markers denote chronological data points for each session. The box represents the extent between 25th and 75th percentile from the median; the whiskers extend to the minimum and maximum values.

A closer look at individual sessions in Figure 6 shows a considerable trend in decreasing of the wrist lift in each set. The results indicate that the subject may have fatigued towards the end of the set. Similarly, in-session trends can be analyzed for the exercise repetition times. Specifically, the participant took longer to complete the first few arm lifts as compared to the last set in almost all of the exercise sessions. The decline in the wrist lift decrease and the repetition times (i.e., smaller variance in data points within a session) indicates some overall improvement in the endurance levels.

C. Evaluation of External Outcome Measures

Table III presents the pre- and post-intervention scores for the external outcome measures introduced in Section III-C as well as the participant demographics. We see that participants #2 and #5 performed at least the same between their pre- and post-intervention scores for all tests. However, participants #3 and #4 showed decline in ACT and 2MSPT while participant #1 showed decline in 2MSPT and 4FUGT, and participant #6 showed decline only in 2MSPT. We observe that 2MSPT scores for most of the participants deteriorated in the post-intervention assessment. It is also important to note that BBS significantly suffered from the ceiling effect as 4 out of 6 participants achieved the perfect (or almost perfect) scores in both pre- and post-intervention tests. In this paper we defer from performing further statistical analysis of the external outcome measures due to a small sample size, short duration of the study, and relatively open exercise regimen that the participants followed.

TABLE III.

Pre- and Post-Intervention Scores

Participant			BBS (score/56)		CST (# stands)		ACT (# curls)		2MSPT (# steps)		4FUGT (sec)
ID	Sex	Age	Pre	Post	Pre	Post	Pre	Post	Pre	Post	Pre	Post
1	F	79	45	48	8	12	14	16	72	47	6.0	8.4
2	M	75	55	55	12	15	16	21	100	107	7.0	6.5
3	M	91	48	52	8	9	15	14	64	50	10.0	7.8
4	F	74	56	56	9	10	16	14	112	107	6.9	6.9
5	F	80	55	56	10	13	15	15	82	99	9.6	6.5
6	F	80	55	55	11	13	16	16	130	102	5.5	4.5

Notes:

F: Female

M: Male

BBS: Berg Balance Scale

CST: Chair Stand Test

ACT: Arm Curl Test

2MSPT: 2-Minute in Place Step Test

4FUGT: 4-Foot Up and Go Test

V. Discussion - Lessons Learned

In this section we discuss usability issues and lessons learned from the reported pilot study. The summary of our findings can be found in Table IV.

TABLE IV.

Summary of Key Findings in the Pilot Study

Category	Benefits	Limitations	Implications
Living Space	• Participants liked the fact that they could exercise at home.	• Only few participants had the required 8’+ space in front of their computer. • Accuracy of tracking was affected in some exercises due to small space. • Chair or other objects were some-times interfering with tracking.	• Kinect 2 has improved skeletal tracking and wider field of view (60Deg HFOV vs. 43Deg HFOV). • Set up the system in a living room or common space for multiple users.
Computer System	• All-in-one computer was easy to set up by a non-technical person, i.e., health coach.	• It was difficult to find space for the computer since it was rather large and bulky. • Participants thought the all-in-one computer would not be appropriate as a long-term solution. • 23" screen was still too small for some older adults.	• Use of a small-desktop PC connected to an existing large screen TV set. • Configure the computer to work as a turn-key system.
User Interface	• Participants found the exercise software easy to use. • Participants were able to use wireless keyboard and mouse successfully.	• Several participants still had difficulty seeing text on the screen.	• Improvements in the UI: larger fonts, buttons with familiar icons, more contrast in colors. • Better use of text-to-speech to assist users in reading larger amounts of text. • Use of dedicated remote to control interaction.
In-Exercise Feedback	• Participants liked the video guidance through the exercise sessions. • Most participants liked receiving encouraging feedback from the exercise system.	• Participants questioned when the system is able to see them. • Participants did not understand meaning of performance measures.	• Adding feedback with the Kinect body tracking information (e.g., human figure, skeleton). • Simplified feedback through repetition counting. • Provide more effective feedback to correct improper performance. • Development of more intuitive performance scores (e.g., scores related to balance, strength, flexibility, endurance).
Exercise Regimen	• Participants in general liked included exercises.	• Some participants found exercises too easy for their level of fitness. • Some exercises were consistently performed incorrectly. • Participants asked to have more variety and control over the exercises for longer-term use of the exercise system.	• Provide larger variety of exercises with different difficulty levels for different parts of the body. • Provide participants with a weekly schedule of exercises based on their past performance.
Outcome Measures	• Kinect-based internal outcome measures proved to be robust enough to perform exercise tracking in most cases.	• Proposed outcome measures for tracking the exercises were difficult to convey to the participants. • External outcome measures (e.g. BBS, SFT) suffered from ceiling effect.	• Derive performance measures based on standard components of physical fitness from Kinect data. • Select appropriate clinical measures based on participant fitness levels to avoid the ceiling effect. • Use of process evaluation methodology [33] to assess effectiveness of intervention.

A. System Setup

Since the system was aimed to be used in a home setting, the main usability limitation was imposed by the Kinect camera. The field of view of the Kinect is 43° × 57° (vertical × horizontal) with the depth range between 0.8 m and 4 m. To capture full body, the user had to be standing or sitting between 1.8 m to 3.0 m away from the camera.

To determine the availability of the space and to examine other constraints in homes of potential users, we had at the start of this project surveyed the living accommodations of 30 participants in the ORCATECH Living Lab program. The survey included information on the type of home, location of the main computer, size of computer monitor, available space in front of the computer desk, type of chair, availability and type of TV screen, and available space in front of the TV screen. The initial survey showed that majority of participants had a desktop computer with a monitor size between 14” and 17”. The computer was typically located either in the office space or a bedroom with 1.2 m to 2.5 m (about 4’ to 8’) available in front of the computer. 12 participants reported to have a newer or flat screen TV in their living room with sizes ranging from 24” to 52”. The participants reported to have anywhere between 1.0 m and 2.5 m (about 3’ to 8’) of free space available in front of the TV screen; however, some participants stressed that they would not want to rearrange furniture to accommodate the system. We have also determined that almost all of the participants had older computers with operating systems that would not be compatible with the Kinect camera. Based on this survey and careful evaluation of different options we decided to use an all-in-one computer with 23” monitor that would provide more flexibility in terms of system setup in different rooms while providing smaller footprint and large display. We set up two identical systems that were moved between participants’ homes to collect the entire dataset.

From the deployment in the six homes we found that the allin-one computer was initially very easy to setup by the coach, however, in some of the homes, it was difficult to find space for the system as it was somewhat heavy and bulky. Most participants had the computer placed next to their existing computer system. One participant used a cart with wheels to set up the computer screen closer while keeping sufficient distance for the Kinect camera in order to view the screen better. The participants reported that although they did not mind having the system in their home they did not see it as a viable long term solution. They expressed preference to use the exercise system either with an existing computer at their desk or having it connected to their TV. At the start of the study, however, majority of the participants did not have newer TV sets or sufficient space in their living room that could accommodate the system.

B. Kinect Tracking

Our exercise system relies on the pose estimation given by the Kinect software. The Kinect tracking algorithm, however, was primarily designed and trained for in-game interactions with several assumptions, such as users are standing, the view is unobstructed, body limbs are away from the trunk, there is no object interaction etc. In our previous publication [30] we examined the accuracy of the Kinect when tracking six of the exercises included in this study. Through the comparison with an optical motion capture system, we determined the distribution of joint position accuracy in three different orientations with respect to the camera. Based on these findings, we determined which of the joints were more reliable to track in particular exercises. The highest errors were found in hip and ankle joints, in particular during the sitting exercises. Similarly from these pilot data, we observed that the sitting skeleton often had much larger errors in the lower parts of the body, especially in the hips. The accuracy of the sitting pose was also influenced by the position of the Kinect camera. If the camera was positioned too low and was thus unable to observe user's thighs, the estimate of the hip position was less accurate. These errors also increased when a chair with larger footprint was used. A more stable skeleton was observed when subjects were performing standing exercises. However, some of the exercises had to be performed while holding on to a chair which sometimes also influenced the tracking accuracy.

Another issue which occasionally prevented participants from being able to complete the set of exercises was faulty user identification. The Kinect keeps track of users identities (by assigning a user ID number) while they are in the scene. If the user leaves the scene or is fully occluded by another user, the tracking system will assign him/her with a new ID number. Our system assumed that the user who is the closest to the camera was the active user whose skeletal data were recorded. Although the participants exercised alone, there were several instances where a piece of furniture was misidentified as a human user and tracked instead. Since the users had no feedback on what is being tracked by the system, their activity was not counted for the particular session. When analyzing such data, we also noticed that when two skeletons are tracked in the scene, regardless of which one is used for the measurement, the stability of measured joint positions is decreased.

When selecting the exercises for this study, we performed extensive testing in the lab [30] to determine which measurements are the most reliable under given conditions. Although it was not possible to achieve stable full-body skeleton tracking in some of the exercises, the extracted measures were sufficiently reliable to track exercise repetitions and trigger corrective cues.

C. Exercise Selection

For the pilot study we used a set of 12 exercises which included activities of upper and lower extremities, primarily focusing on improving balance and strength. The exercises were chosen with regard to safety considerations (since the participants were performing the exercises without direct supervision) and the ability of the Kinect to robustly recognize and track the user performing a particular exercise.

The most challenging exercise was Cops and Robbers which consists of three stages: extension of arms, retraction of arms, and rotation of forearms to vertical position. During initial testing, subjects were typically combining the second and third stage, therefore the system was not able to properly count the repetitions. After discussing the issue with the coach, we decided to modify the requirements of this exercise and define only two stages which helped the participants get their repetitions counted even when performance was only partially correct. We believe it is necessary to provide clearer reminders of particular steps before and during the exercise.

Overall, the participants liked the selection of exercises, although some of them felt that the exercises were not challenging enough and did not have sufficient diversity to use this system for a longer time period. Other participants reported that ‘the exercise made them sore’ as they exceeded their regular level of activity. In the future, we thus plan to include broader selection of exercises that would target different levels of users, from easy to more challenging, and provide workout for different parts of the body, such as upper, lower, and full body. To compensate for the limitations of the Kinect to track the human body in more challenging poses, some of the exercises could provide only video instructions while no real-time detection of exercise stage is performed.

D. In-Exercise Feedback

When designing the in-exercise feedback screen, we tried to incorporate familiar elements of traditional workout videos as opposed to using 3D graphics that is typical for video games in order to appeal to older adults. The feedback on the exercise performance was overlaid on top of the video layer in form of performance bars, repetition counter, and textual cues. Additional audio cues were played along side the textual messages.

The participants reported that they found the in-exercise feedback screen useful, although some of the participants did not understand how the performance bars related to their movement. Some also did not remember that the thick mark inside the performance bar represented their last maximal performance which they were supposed to reach or exceed. The subjects liked the audio cues that were provided by the system, for example to remind them to sit tall during a particular exercise or to encourage them. One of the participants, however, mentioned that ‘the kudos and feedback were not him’ and he did not think they were necessary.

The participants liked the instructor. They mentioned that ‘the instructor talks clearly, speaks to the camera’. They found the demonstrations of each exercise to be useful for correct performance. The subjects, however, complained about the readability of the textual information provided during the exercise as it was too small for them to read from such a distance. This issue could be addressed by reducing the amount of textual information and use of a bigger screen, such as a TV screen.

Although the participants were able to see the output video captured by the Kinect camera during their exercise performance, they could not tell if the system was tracking them correctly or not. As discussed earlier, some of the subjects were not tracked properly or their repetitions were not always counted. We initially decided to provide only the video feedback in order to minimize the complexity of the visual feedback. In the future, we plan to include more focused feedback on the data output from the Kinect (e.g., segmented human figure) that would provide better cues to users to be able to self-correct some of the camera/tracking related issues.

E. User Interface

For the interaction with the exercise software we implemented several options, such as a mouse, keyboard shortcuts, touch screen, and voice control. During our initial testing we also considered use of a wireless presentation remote, however, we decided against it as it required turning it off after each use, otherwise the battery life was significantly reduced. We also determined that the voice control would be too challenging for the participants to use it reliably, therefore it was turned off for the pilot study. We eventually opted for using wireless keyboard and mouse for interacting with the software since all the participants in our pilot study were computer users and were thus already accustomed to the keyboard/mouse. Keyboard shortcuts (i.e., single key strokes) or mouse clicks were used to interact with the on-screen buttons. Overall the subjects reported that ‘the interaction with the screens was very intuitive and they were able to easily find their way around the program’. One of the participants reported that ‘the text on the buttons was difficult to read’, however most participants were able to read the button labels. With increased complexity of the exercise software that would in the future offer more variety in the selection of exercises, the user interface will require use of symbolic buttons, color cues, and less textual information.

F. Outcome Measures

As discussed previously, the internal and external outcome measures need to be able to detect changes in user performance that are associated with the standard components of the physical fitness, such as balance, flexibility, strength, and endurance. The quantification of these are important in order to guide the health coach as well as to provide feedback to the users which will incentivize them to exercise more regularly.

In our pilot study we computed several high-level performance primitives from the Kinect skeletal tracking as the internal outcome measures to evaluate user performance and to provide in-exercise feedback, stage detection, repetition counting, and exercise correctness verification. These measurement primitives proved to be robust enough, however, for each exercise, the selection of the appropriate measures had to be determined manually in collaboration with an exercise consultant. There is a need to develop a more generalized framework that would define appropriate measures for different exercises in more automated manner. Another open issue relates to the interpretation of the Kinect-derived performance measures in terms of aforementioned standard components of the physical fitness.

The effects of the exercise intervention also need to be interpreted in terms of standardized clinical measures, as those are the most useful for the health coach. The subjects in our pilot study were evaluated by several different scales at the beginning and the end of their exercise regime. Although we did not find a significant correlation by direct comparison of our internal outcome measures against the external outcome measures, we observed some trends that could be explored further with the help of more sophisticated approaches and data collected over a longer time period. For instance, instead of using primary measurements directly as the internal outcome measures, one could extract more advanced parameters that could consider a subject's simplified dynamics model (e.g., mass and inertia) to relate the performance to the standard components of physical fitness and provide a better foundation for direct comparison between such internal measures and the external outcome measures.

For the future studies, selecting the appropriate clinical tests will be as important as selecting the appropriate exercises for our coaching system. For this pilot study, we administered BBS and SFT. The BBS was developed originally to measure balance among older adults who need assistance in ADLs and live in residential care facilities. However, all participants in our pilot study were living independently. Therefore, the BBS scores suffered significantly from the ceiling effect as 4 out of 6 participants achieved the perfect (or almost perfect) scores in both pre- and post-intervention tests, leaving no room to detect potential improvements. Alternatively, to better account for cases where standard clinical tests fall short of measuring changes successfully, a comprehensive process-evaluation plan [33] can be developed to assess the overall implementation of the proposed exercise coaching system.

G. Social Aspects

Another important factor influencing exercise adherence is the social aspect. Social support can in general increase the self-efficacy and subsequently enhance adherence [34]. In our study, we initially discussed various possibilities to include social interaction between the participants, such as having the exercise system in a common space for different users to use, providing ability for participants to share their scores with others, forming a competitive environment, etc. However, we had decided to exclude that component in the first iteration of the exercise system as we did not yet fully understand the potential technical difficulties (e.g., wireless internet connectivity) and participants’ interest in such interactions. The social interaction with regard to the exercise system was thus primarily limited to the weekly communications with the health coach. We will, however, explore some of the aforementioned social elements in our future work.

VI. Conclusion

Interactive exercise systems hold promise in promoting physical activity for all ages as shown by several previous stud ies. One of the interesting questions remains on how to balance the attractiveness and effectiveness of the exercise systems in real-world settings while considering various factors, such as age and educational level of users, system and environmental constraints, type of feedback, level of gamification, incentives, and social interaction. In our pilot study we deployed an interactive exercise system with automated feedback and coaching capabilities for older adults in their homes. Physical performance metrics and inference algorithms were based on data from the Microsoft Kinect camera mounted in users’ homes. In this paper we showed that despite several real-world challenges, the system was able to collect continuous exercise data during the entire deployment period while the participants expressed positive attitude towards using such a system for longer term in the future. Although this study was limited in terms of number of subjects and duration of the trials, it provided several insights on the system deployment and usability issues concerning elderly users. In our future work we are planning to improve the software capabilities with a wider selection of exercises, more focused and relevant in-exercise and post-exercise feedback, and a more flexible user interface. This will provide a foundation for a subsequent more controlled research study with a larger number of participants studied over a longer period of time, with a goal of understanding how best to help older adults achieve improved physical functioning and maintain independence.

Biography

Ferda Ofli (S’07-M’11) received B.Sc. degrees both in Electrical and Electronics Engineering and Computer Engineering, and the Ph.D. degree in Electrical Engineering from Koç University, Istanbul, Turkey, in 2005 and 2010, respectively. He is currently a postdoctoral researcher in the Teleimmersion Laboratory at the University of California at Berkeley, Berkeley, CA. His research interests span the areas of multimedia signal processing, computer vision, pattern recognition and machine learning. He received Graduate Studies Excellence award in 2010 for outstanding academic achievement at Koç University.

Gregorij Kurillo received the B.Sc. and Ph.D. degrees from School of Electrical Engineering, University of Ljubljana, Slovenia, in 2001 and 2006, respectively. He was a Research Assistant with the Laboratory of Robotics and Biomedical Engineering at the same institution from 2002 to 2006. From 2006 to 2009 he was a Postdoctoral Researcher and since 2009 the Head Research Engineer at the Teleimmersion Laboratory at University of California at Berkeley. He currently also holds a shared research appointment with Department of Physical Medicine and Rehabilitation, University of California Davis. Dr. Kurillo's research interests include camera calibration, 3D vision, image processing, robotics, virtual reality, and rehabilitation engineering.

Štěpán Obdržálek received his MS degree in computer science and PhD degree in computer vision from the Czech Technical University in Prague. Since then he worked as a research fellow at the Centre for Machine Perception, CTU Prague, and at the University of California, Berkeley. His research interests include feature based object recognition and localization, textureless object recognition, intelligent vehicles, 3D motion analysis and egomotion estimation.

Ruzena Bajcsy (LF’08) received the Master's and Ph.D. degrees in electrical engineering from Slovak Technical University, Bratislava, Slovak Republic, in 1957 and 1967, respectively, and the Ph.D. in computer science from Stanford University, Stanford, CA, in 1972. She is a Professor of Electrical Engineering and Computer Sciences at the University of California, Berkeley, and Director Emeritus of the Center for Information Technology Research in the Interest of Science (CITRIS). Prior to joining Berkeley, she headed the Computer and Information Science and Engineering Directorate at the National Science Foundation. Dr. Bajcsy is a member of the National Academy of Engineering and the National Academy of Science Institute of Medicine as well as a Fellow of the Association for Computing Machinery (ACM) and the American Association for Artificial Intelligence. In 2001, she received the ACM/Association for the Advancement of Artificial Intelligence Allen Newell Award, and was named as one of the 50 most important women in science in the November 2002 issue of Discover Magazine. She is the recipient of the Benjamin Franklin Medal for Computer and Cognitive Sciences (2009) and the IEEE Robotics and Automation Award (2013) for her contributions in the field of robotics and automation.

Holly Brugge Jimison received her PhD in Medical Information Sciences from Stanford University in 1990. She is currently a professor of practice in the College of Computer & Information Science and the Bouvé College of Health Sciences at Northeastern University in Boston. She directs the Consortium on Technology for Proactive Care with research projects on computational modeling of patient data from sensors in the home and environment, as well and technology for health coaching interventions. Dr. Jimison is a Fellow of the American College of Medical Informatics.

Misha Pavel is a Professor of Practice jointly appointed between College of Computer and Information Sciences and the Bouvé College of Health Sciences at Northeastern University. Dr. Pavel came to Boston from a position of a Program Director of Smart and Connected Health on a leave from at Oregon Health and Science University where he was a professor at the Department of Biomedical Engineering, with a joint appointment in the Department of Medical Informatics and Clinical Epidemiology. He is also a visiting professor at Technical University of Tampere. Previously he served as chair of the Department of Biomedical Engineering (he founded in 2001) and as Director of the Point of Care Laboratory, which focuses on unobtrusive monitoring, neurobehavioral assessment and computational modeling in support of healthcare, with a particular focus on chronic disease and elder care. His earlier academic appointments included positions at New York University and Stanford University. In addition to his academic career, Professor Pavel was a member of the technical staff at Bell Laboratories in early 1970s, where his research included network analysis and modeling, and later at AT&T Laboratories with focus on mobile and Internet-based technologies. His current fundamental research is at the intersection of multilevel computational modeling of complex behaviors of biological and cognitive systems, and augmented cognition. His most recent efforts are focused on fundamental science and technology that would enable the transformation of healthcare to be proactive, distributed and patient-centered. He has a Ph.D. in experimental psychology from New York University, an M.S. in electrical engineering from Stanford University, and a B.S. in electrical engineering from the Polytechnic Institute of Brooklyn. Misha Pavel is a Senior Life Member of IEEE.