ToPick: Time-of-Pickup Measurement for the Elderly using Wearables

Abstract

The ability to pick up objects off the floor can degrade over time with elderly individuals, leading to a reduced quality of life and an increase in the risk of falling. Healthcare professionals have expressed an interest in monitoring the decline in pickup ability of a subject over extended periods of time and intervening when it becomes hazardous to the subject’s health. The current means of evaluating pickup ability involving in-clinic patient visits is both time and financially expensive. There is a clear need for a cost-effective, remote means of pickup evaluation to ease the burden on both patients and physicians.

To address these challenges, we introduce a Time-of-Pickup (ToP) solution, called ToPick, designed for the automatic assessment of pickup ability over time. The practical performance of ToPick is evident, demonstrated by a minimal median error of approximately 100 milliseconds in evaluating 20 pickup events among 10 elderly individuals. Furthermore, ToPick exhibits a high level of reliability, achieving perfect accuracy, precision, and recall scores for pickup event detection. We actualize our research findings by designing an application intended for adoption by both healthcare practitioners and elderly individuals. The app aims to reduce both time and financial costs while enabling mobile treatment for users.

I. Introduction

Senior citizens face challenges in picking up objects in their daily routines. A failed pickup attempt may result in a fall while reaching for an object [1]. Falls for elderly individuals have significant consequences such as bone fractures or head injuries. These traumatic consequences can result in a loss of independence and tragic socioeconomic and health-related ripple effects for seniors who are subsequently hospitalized [18]. Falls resulting in hospitalizations are common. Every year, nearly 3 million seniors visit the emergency room because of a fall [2]. Healthcare practitioners such as physical therapists want to prevent falls by monitoring an individual’s ability to pick up objects over an extended period (e.g., months), and intervene if there is a problematic decline. The early identification of symptoms indicating a degrading pickup ability can help prevent injuries. Clinicians can prescribe treatments such as physical therapy and take action to prevent falls.

Current methods for assessing pickup ability over time require patients to visit a clinic and execute a series of pickup tasks on a pressure-sensing floor mat such as the ZenoMat [6] or GaitMat [8]. A test such as the Berg balance test [19] assesses subject health by asking a subject to reach down and pick up an object from the ground. A skilled practitioner analyzes this data to determine pickup time and “grade” the subject’s health. The process is repeated at regular intervals (e.g., every month) at a specific mat-equipped location.

There are pain points associated with the current method. First, it involves the use of expensive hardware, the patient’s time, and the skilled practitioner’s time. Secondly, it lacks portability, requiring the patient to visit the clinic and consequently imposing time and geographical restrictions. Thirdly, labeling pickup time is laborious due to the inability of the 2D pressure mat to distinguish between real steps, foot drags, small shuffles, etc. To solve these pain points, we ask: How can we automatically measure the time for a subject to pick up an object from the floor with wearables?

To answer this research question, we develop a Time-of-Pickup (ToP) solution, called ToPick, that can automatically measure pickup events in any location with any non-skilled user. ToP describes the date and time when pickup events occurred, as well as how long it took for the subject to complete each of them. The solution involves recording pickup events using an app, extracting features, modeling pickup events, pruning false positive events, and returning the real pickup event measurements to the user in the app. The results show that ToPick is perfect for our problem scenario and can serve as a replacement or augmentation for the current expensive methods used by practitioners.

We make the following contributions in this paper:

1. We create ToPick, a solution that can detect and measure pickup events. We design this algorithm using one patient’s data and show that our solution generalizes to the rest of our 10 subjects.

2. We demonstrate that ToPick’s median measurement error is just 100 ms when compared to the ground truth when evaluating it on elderly individuals’ pickup data.

3. We conduct a user study on 10 elderly patients who performed 38 pickup events, giving us rich ground truth data which helps us assess ToPick’s correctness.

The remaining sections of this paper are organized as follows: Section II discusses related works. Section III provides details on the user study conducted with 10 elderly individuals who participated in 38 pickup events. Section IV outlines the design of ToPick and its ability to measure pickup events. In Section V, we present the performance evaluation, demonstrating low measurement error and perfect accuracy, precision, and recall. Finally, Section VI summarizes our research findings and highlights potential future research.

II. Related Works

Pressure sensing is used to assess gait-health by practitioners. The GAITRite [9] pressure mat sensor creates a map of the subjects’ footsteps. This map can then be analyzed to infer pickup time and other gait attributes. These pressure sensors have a major limitation: the data is in two dimensions. Since the mat does not know how high the foot was lifted from the mat, it is tricky for the practitioner to distinguish between real steps and small foot movements. We seek to use vertical axis information and also offer more portable than the mat.

IMU sensors are a popular choice for both clinical and real-world gait analysis [3]–[5], [7], [10]–[17], [21], as they can be used to accurately detect normal footsteps and small movements, identify stride parameters such as stride velocity, and as well as diagnose abnormal gait [20]. These sensors can be combined with mobile gait analysis healthcare solutions [7], [12], [14] to facilitate the monitoring of subject health in a non-clinical environment. Since only ToPick supports pickup measurement grading over time, our contributions are unique.

III. User Study Data Collection

A. Hardware Description & Time-series Data

The Ultigesture (UG) wearable IMU sensor platform [22], shown in Figure 1a, consists of a 3D gyroscope, accelerometer, and magnetometer. The devices cost just $10 to manufacture and consist of a Cortex-M4 processor and a BLE module. We have three of these UG devices: one on each of the leg’s ankles and one on the ground. Figure 1b shows that the vertical axis is different depending on the setup orientation of the sensor. The ankle-mounted sensors are attached to the subject’s ankles with velcro straps and will monitor the subject’s foot movement. The subject picks up the sensor on the ground. The sampling rate is 100Hz. We collected pickup event data from 10 participants with diverse health conditions between 75 and 87 years old under an approved IRB protocol.

Fig. 1:

UG.

B. Ground Truth 1: Detection

All 38 pickups have detection ground truth as indicated by the ground IMU device. One pickup event can be seen in Figure 2. The resting value of the ground-IMU is non-zero, due to Earth’s gravity (~9.8 m/s²). When the total accelerometer value spikes above the resting value in our time-series data, it indicates that it has been “contacted” (i.e., physically moved) by the subject. When this accelerometer value pattern switches from constant to increasing (the first spike in data), we mark a pickup event detection ground truth. A pickup span is defined as all of the time-stamps in between the start and end time-stamps of a pickup event. ToPick’s pickup span estimations must include one of the 38 ground truth timestamps to be valid.

Fig. 2:

Ground IMU is contacted by subject.

C. Ground Truth: Duration

20 out of 38 pickups with videos are used for duration ground truths. The videos were recorded at 60 Frames Per Second (FPS). Two pickup events were recorded on video for each of the 10 patients. These were the first and final pickup events that the subjects performed. We cannot determine the duration Δ error for the remaining 18 pickups, since we do not have videos for them. The three images in Figure 3 depict the start, contact, and end frames of a pickup event respectively. We manually label the start and end frames for the first and final pickup events for each subject by analyzing videos of 20 pickup events. The contact frame will synchronize to either the first or last spoon UG’s contact timestamp in the event detection ground truths. We define the pickup start as the moment of the beginning of the trunk bending towards the floor/one foot is no longer advancing. We define the pickup end as the start of the first step after picking up an item. To calculate the true duration of the pickup using our video ground truth frames: $\text{Total Time}(\text{ms})=(En{d}_f-{\mathit{\text{Start}}}_f)\times \left(\frac{1}{\text{FPS}}\right)\times 1000$.

Fig. 3:

Start, contact, & end frames of a pickup.

IV. Topick System Design

A. ToPick System Overview

In order to answer our research question, we present a ToP measurement system called ToPick. The system was designed using one subject’s data. The high-level overview of ToPick is provided in Figure 4. First, data is first collected by the app following a user story. Secondly, the data is sent for feature extraction. Thirdly, pickup spans are modeled using a rule-based algorithm. Fourthly, we eliminate false positives using pruning steps. Finally, the ToP measurement results are returned output by the app and are interpreted by a user.

Fig. 4:

ToPick System Overview.

Data Collection User Story:

To facilitate the recording of pickup events, the subject has access to a mobile device with the ToPick app. The subject is equipped with two UG sensors, one on each ankle. They are attached with Velcro bands. The two UG sensors send accelerometer data to the app when connected. The subject starts recording data by pressing a button on the app. The subject performs the protocol: (1) walks forward for a few normal steps, (2) picks up an object, and (3) continues walking regularly for a few steps. The patient may repeat the actions (1–3) as many times as desired. The subject stops recording data by pressing a button on the app.

B. Feature Extraction

Once data has been collected, ToPick extracts meaningful features that help it determine pickup event measurements. Data is first smoothed to eliminate noise and facilitate analysis, then it is scaled to account for different hardware calibrations, and then it is summed between the left and right feet’ UG IMU devices to see the total movement between both feet. The following subsections describe each of these steps.

1). Selection of Vertical Axis:

Accelerometer y-axis (vertical) data is used as an indicator for a pickup event start for the feet. The vertical axis is chosen since we hypothesized that the vertical axis will have the most identifiable pattern to distinguish between real steps and the foot movement that indicates a pickup event start. Consultation with a healthcare practitioner concerned with our domain problem revealed that there is a distinct step pattern that indicates the start of a pickup event. This pattern should have a smaller vertical accelerometer peak corresponding to a small footstep at the start of a pickup event. This is a smaller peak value than during normal walking. The unprocessed vertical accelerometer value for both UG IMU devices on the feet can be seen in Figure 5.

Fig. 5:

Accelerometer time-series before any processing.

2). Vector Magnitudes & Smoothing:

We calculate vector magnitudes to standardize our data. We also smooth the data using Butterworth filtering to help with pattern recognition. The smoothed vector magnitude vertical accelerometer value for both UG IMU devices on the feet can be seen in Figure 6.

Fig. 6:

Accelerometer time-series after smoothing.

3). Scaling:

To mitigate the effects of different hardware calibrations, we scale data for each UG sensor using the min-max readings. The new scale is 0–1. For each element i in our data: ${\text{scaledValue}}_i=\frac{x_i-\text{min}}{\text{max}-\text{min}}$. The scaled vertical accelerometer value for both UG IMU devices on the feet can be seen in Figure 7. Note that at this point the y-axis units are no longer m/s² due to our scaling function.

Fig. 7:

Accelerometer time-series after scaling.

4). Total Vertical Foot Movement:

To help us detect the start of a pickup event, we want to extract a new data column that illustrates the total y-axis foot movement at any point in time, between both feet. To do this, we sum the total y-axis acc vec mag between the left and right UG sensors. We chose to use 50ms “bucket” interval sizes in ToPick to sum the total y-acc value between both feet. The summed scaled vertical accelerometer value for both UG IMU devices on the feet can be seen in Figure 8. The scale is now between 0 and 2, since we have summed two time-series with a scale between 0 and 1. Bucket “T” is the interval which the data is summed within.

Fig. 8:

Accelerometer time-series after data extraction.

C. ToP Modeling

Pickup spans are modeled using a rule-based algorithm. ToPick analyzes the time-series created during feature extraction and plots suspected pickup span measurements. There are two general steps included in this algorithm. First, ToPick calculates three tactical thresholds. Secondly, ToPick “walks” along the time-series data and marks span start and endpoints.

• Common Line: Allows determination of when both feet are in a fixed upright position with no movement. It is considered as the resting value of both feet’s vertical accelerometer, calculated by determining the most frequent value in the dataset. It is not null, due to Earth’s gravity.

• Upper Threshold: Calculated by adding an α value to the common “resting” value. This threshold is used to mark the start and end points of a pickup span, indicating when the feet are leaving or entering a vertical accelerometer value close to the common value.

• Healthy Step Threshold: Calculated by multiplying the “maxFound” by a tunable β parameter. maxFound is the maximum value found in the combined vertical accelerometer time series for that subject. Small foot movements, such as shuffles or foot drags, will show a spike in vertical accelerometer value but will not reach the level of a healthy step. A foot movement spike that has values exceeding this threshold can earn “step credit”. This value is used in Pruning rule # 2.

The thresholds permit ToPick’s rule-based decision model. We start a pickup span at the timestamp where the combined feet’ accelerometer value descends below the upper threshold. This will correspond to the practitioner-labeled pickup start when the trunk of the subject is bending towards the floor, and one leg has stopped moving. We end a pickup event span at the timestamp where the combined feet’ accelerometer value ascends below the upper threshold for the second time. This is the moment when the subject starts to take a real step after a pickup contact has occurred when the subject touches the UG IMU on the ground. We end a pickup span early if the combined feet’ accelerometer value reaches the health step threshold. This early exit condition signals that a regular step is taking place, and the pickup pattern has not been followed.

The threshold calculations, the pickup pattern, and the “walk” algorithm’s rules are visualized in Figure 9. The pink dotted line shows the pickup pattern that is our universal indicator. The early-exit condition spans are marked with a red star. After the thresholds have been calculated, and the walk has been completed, ToPick produces results.

Fig. 9:

(c) “ToP Modeling” as shown in the System Overview.

D. Pruning

We now must remove all the false positives so that we only give estimates of real pickup events to the end-user. Each stage of pruning is detailed by a plot. The pickup spans subject to removal by the next rule are encompassed in a dotted pink box. The initial set of ToP estimates is shown in Figure 10.

Fig. 10:

The original set of ToP estimates.

1). Rule #1 (Short Spans):

We disregard any span that is under 800ms as a false positive. These erroneous estimates result from the early exit condition described in Figure 9. The spans marked by a red star are too short to be pickup events. Figure 11 shows the resulting span set after this rule.

Fig. 11:

The set of ToP estimates after rule #1.

2). Rule #2 (Spans out of Context):

We disregard any span that is out of the context of a pickup event. Since we are measuring pickup ability in between walking periods, we only keep pickup estimate spans that have start and end points in close temporal proximity to steps with “step credit”. There must be a “healthy” step within 1000ms prior to the pickup start time, and within 3000ms after the pickup end time.

3). Rule #3 (Long Span Outliers):

Finally, we disregard any pickup span that is over 5 seconds. These are likely caused by the subject placing down the spoon UG sensor after performing a pickup. The patient would then stand still and catch their breath before continuing. The resulting set of ToP estimates after rule #3 is shown in Figure 13. Once the pickups have been processed by ToPick, the results are returned on the app’s analysis screen.

Fig. 13:

The set of ToP estimates after rule #3.

V. Performance Evaluation

A. Settings

We compare ToPick’s performance against a naive algorithm. We create an in-house naive algorithm since we were not able to find a state-of-the-art algorithm specifically for pickup detection in recent literature. The naive algorithm detects periods when the subject is standing still by analyzing the vertical movement between both feet. This algorithm is considered naive since it assumes that each subject will plant both legs firmly on the ground, pick up an object, and then resume regular walking.

B. Detection Accuracy

We determine detection performance by looking at how many of the 38 pickups included actual pickup contact timestamps (when a subject touched the spoon UG device). The ToPick algorithm has perfect accuracy, precision, and recall as shown in Figure 14. The naive algorithm has an acceptable precision of 0.96 but has lackluster accuracy and recall of 0.46 and 0.45 respectively. These results are the result of numerous false negatives produced by naive.

Fig. 14:

Detection accuracy (38 pickups).

C. Duration Error

We evaluate ToPick’s total pickup duration measurement by comparing ToPick’s pickup estimate against the ground truth pickup time determined via video analysis, and calculate the difference Δ. Figure 15 shows that ToPick’s median error is just 100ms, which is 4x less than naive’s 400ms. The mean error is 150ms for ToPick, which is also nearly 4 times less than the naive error of 582ms. ToPick outperforms the naive algorithm. Figure 16 shows a comparison in duration error for each pickup. Naive gave false negatives for S1 E1, S2 E4, S6 E1 & E4, S7 E1 & E4, S8 E4, S9 E1 & E4. All of these results are indicated by a single asterisk. S3 was a challenge for both ToPick and naive. This subject uses a four-wheeled walker to help them walk and perform pickups, meaning that they are an outlier in our patient data. Without the outlier, the mean delta error for ToPick to just 100ms. Given these results, we can say that ToPick is successful in its mission of accurately and automatically measuring pickups.

Fig. 15:

Duration error (20 pickups).

Fig. 16:

Duration error (20 pickups).

VI. Conclusion

We answered our research question by making three contributions. First, we conducted a user study with an elderly population. Secondly, we designed the ToPick system and algorithm. Thirdly, we showed that ToPick is accurate and has a low median error of just 100ms when evaluating performance across 20 pickups. There are no false positives. Our future work includes the deployment of ToPick in a validation study.

Fig. 12:

The set of ToP estimates after rule #2.