Automated Video Exposure Assessment of Repetitive Hand Activity Level for a Load Transfer Task

Abstract

Objective

A new method is described for automatically quantifying repetitive hand activity using digital video processing.

Background

The hand activity level (HAL) is widely used for evaluating repetitive hand work. Conventional methods using either a trained observer on or off site, or manual off site video analysis, are often considered inaccurate, cumbersome or impractical for routine work assessment.

Method

A cross correlation-based template-matching algorithm was programmed to track the motion trajectory of a selected region of interest over successive video frames for a single camera to measure repetition frequency, duty cycle and HAL. A simple paced load transfer task was used to simulate a repetitive industrial activity. Twelve participants were videoed performing the task for varying HAL conditions. The automatically predicted HAL was compared to the manually measured HAL using frame-by-frame video analysis.

Results

Predicted frequency, duty cycle and HAL were in concert with the manually measured HAL conditions. The linear regression slope of the automatically predicted values with respect to the manually measured values were 0.98 (R²=.79), 1.27 (R²=.63), and 1.06 (R²=.77) for frequency, duty cycle, and HAL respectively.

Conclusion

A proof-of-concept for automatic video-based direct exposure assessment was demonstrated.

Application

The video assessment method for repetitive motion is promising for automatic, unobtrusive, and objective exposure assessment, which may offer broad availability using a camera enabled mobile device for helping evaluate, prevent and control exposure to repetitive motions related to upper extremity injuries in the workplace.

INTRODUCTION

Upper extremity injuries are prevalent in repetitive hand intensive work (Fan et al., 2009; Harris, Eisen, Goldberg, Krause, & Rempel, 2011; Hegmann et al., 2009; Silverstein et al., 2010). A variety of observational and self-report methods are available to help assess exposure to repetitive hand motion (Burt & Punnett, 1999; Garg & Kapellusch, 2009; Homan & Armstrong, 2003; Ketola, Toivonen, & Viikari-Juntura, 2001; Paquet, Punnett, & Buchholz, 2001; Spielholz, Silverstein, Morgan, Checkoway, & Kaufman, 2001). Instruments for direct measurement are also available (Buchholz & Wellman, 1997; Jonsson & Johnson, 2001; Marras & Schoenmarklin, 1993; Marshall, Mozrall, & Shealy, 1999; Schoenmarklin & Marras, 1993).

A commonly employed observational method for quantifying the degree of repetition is the Hand Activity Level (HAL) scale, which is based on the 10 point visual-analog scale originally proposed by Latko and Armstrong (1997). It is one of the factors in ascertaining the threshold limit value (TLV) for repetitive manual tasks (American Conference of Governmental Industrial Hygienists., 2009). The HAL scale (Figure 1), ranges from 0 to 10, quantifies repetitive motion ranging from idle hand activity (0) to rapid steady motion with difficulty keeping up (10). It can be determined by a trained observer using the scale in Figure 1 or by a table using objective measurements of frequency and duty cycle from stopwatch time study or frame-by-frame video analysis.

Figure 1.

Hand Activity Level visual-analog scale.

HAL is based on duty cycle and exertion frequency. For many tasks, these are associated with hand and arm movements and static exertions. The frequency is the rate of repetition in cycles per second or Hz. The duty cycle is the percent ratio of hand exertion time to the total cycle time of the activity (i.e. exertion time/cycle time). Consequently a 100% duty cycle refers to repetitive work where force is exerted for the entire time and a 50% duty cycle occurs when force is exerted half of the time.

Several studies have attempted to automate the analysis of repetitive motion measurement in the workplace (Bhattacharya et al., 1999; Person, Hodgson, & Nagy, 2001). Spectral analysis of electrogoniometer data was proposed as an efficient method for quantifying repetitive joint motions that agreed closely with, and was more precise than observational or frame-by-frame analysis (Juul-Kristensen, Hansson, Fallentin, Andersen, & Ekdahl, 2001; Radwin & Lin, 1993; Yen & Radwin, 2000). Radwin et al. (1994) devised an approach analogous to a sound level meter using frequency-weighted filters based on psychophysical data for equivalent discomfort levels resulting from instruments measuring repetitive wrist movements of different amplitudes and frequencies (Lin & Radwin, 1998a, 1998b; Lin, Radwin, & Snook, 1997).

Direct measurements have been mostly limited to research studies, and both observations and direct measurements are often considered impractical for routine industrial analysis. Compared to instruments, observation is non-invasive but lacks precision and accuracy, is not suitable for long observation periods, and requires considerable analyst time (Lowe, 2004). Alternatively, attaching sensors on working hands is time consuming (Yen & Radwin, 2000), and sensors may interfere with normal working operations. Not only is instrumentation use resource intensive, but the required technical knowledge often makes this approach inaccessible to general industry. Considering these limits, the protocols for the NIOSH multi-institutional musculoskeletal research consortium mostly involved indirect observational data related to motion and exertions (Garg & Kapellusch, 2009).

This paper describes a new approach that uses digital video processing to automatically measure repetitive motion exposure using the HAL metric. A direct video assessment method has several advantages over commonly used observational and instrumentation methods. It is objective, can automatically perform the analysis in real-time, requires minimum human intervention, and is determinate for a given video segment.

Despite advancements in image processing, there are no previous developments that address automatic video analysis of repetitive hand activity for occupational health and safety assessment. Related bodies of work are rooted in video tracking, and more recently vision-based control. Traditional methods used for tracking monocular image sequences normally require prior knowledge of a scene (Frade, Marroquin et al. 1997), or complex stochastic cues of the foreground/background model (Sidenbladh and Black 2001). Similar assumptions are impractical and unacceptable for our purposes, since the background activity of occupational settings is usually very busy, and depending on the workplace, may vary considerably. Furthermore the computational complexity of a stochastic model is high, and is not ideal for real-time mobile computing where low power signal processors are needed.

Some previous video-based cyclic motion analysis methods (Allmen and Dyer 1990) require complex analysis and recovery of spatiotemporal (ST) surfaces and ST curves for different scale space. Tsai, Shah et al. (1993) presents an approach to performing cycle detection using autocorrelation and Fourier transform techniques. The approach we developed is made practical by relaxing the need for the high precision normally important for spatiotemporal motion tracking of joints and body linkages, by emphasizing temporal patterns and by adopting a semi-automatic approach whereby the analysts uses human discretion to interactively select a region of interest associated with the repetitive task, such as a hand or arm to track, rather than imposing an a priori model of the tracked activity.

We propose an algorithm that operates by identifying and tracing the pixel pattern that best resembles the selected region of interest as it changes location in successive video frames. The velocity of this pixel area, relative to a stationary pixel region, is used to estimate the rate of repetition and pauses needed to calculate HAL. Consequently the analysis complexity is greatly reduced and may be more tolerable of the numerous variations encountered in field video recordings of occupational tasks.

METHODS

Participants

Twelve young, healthy volunteers were recruited from within the University of Wisconsin–Madison. Their ages ranged from 18 to 32 years. There were 6 males and 6 females, all right-hand dominant, except one male. Before each session was scheduled, a short phone prescreening was utilized to help qualify candidates to ensure that they did not have any ongoing hand/elbow injuries which could prevent them from gripping and transferring weighted bottles for one hour. They were also asked to report basic demographic information. The protocol was reviewed and approved by the Institutional Review Board of University of Wisconsin–Madison prior to subject recruitment and all subjects participated with informed consent.

Apparatus and Experimental Procedures

A laboratory mock-up of a simple load transfer task was implemented in order to simulate repetitive motion activities typically performed in an industrial setting (Figure 2). The loads were equally weighted plastic bottles filled with calibrated quantities of lead shot. The apparatus consisted of a 6 RPM turntable, driven by an electric motor, attached to a chute for the weighted bottles to drop into. The task involved reaching with the right hand for a weighted bottle (2.2 N) in a dispenser located at the end of a chute, grasping and moving it to the turntable, and continuously repeating the operation after the bottles return to the chute. The repetition frequency and duty cycle were paced in order to achieve various HAL levels.

Figure 2.

Laboratory repetitive motion task where a subject gets loaded bottles from the tray (Left) and moves them onto the turntable (Right).

Each subject was required to perform seven paced tasks in one single session. Participants were given a practice period in order to become familiar and proficient with the pace Each task contained 10 to 60 cycles and lasted 60 seconds to 200 seconds, depending on the specific condition. Two minutes of rest were provided in between experimental conditions in order to prevent fatigue. The seven paced tasks that were performed in the laboratory varied in repetition frequency (Hz) and duty cycle (%), in order to achieve HAL levels between one and five. The frequency and duty cycle experimental conditions are described in Table 1.

TABLE 1.

Frequency and duty cycle settings for each HAL condition.

Frequency (Hz)	Duty Cycle (%)	Paced HAL	HAL by Eq. 1
0.125	25	1	0.1
0.250	10	2	2.1
0.250	20	2	2.4
0.500	20	4	3.7
0.500	30	4	4.1
0.500	50	5	4.7
1.000	50	5	5.4

A three-tone set of audio cues were utilized for pacing the task. The first tone signaled to reach, grasp, and move the bottle from the dispenser to the turntable. The second tone signaled the subject to return the hand to the rest position. The third tone signaled the start of a rest period where the subject was instructed to remain at rest until the cycle repeated. Note that for the 50% duty cycle condition (HAL level of 5) only two tones were used since there was no rest period in between cycles.

A JVC Everio GZ-MS230BUS video camera was positioned on the right, three meters away from the subject. Each experiment was videoed and stored as a single MPEG video file with 720 × 480 resolution at the frame rate of 30 frames per second (fps). The videos were each processed by the video algorithm and HAL was calculated based on the movement frequency and duty cycle. Each video was also analyzed manually frame-by-frame using Multimedia Video Task Analysis® software in order to measure the cycle time (i.e. 1/frequency) and hand exertion time, in order to compute frequency, duty cycle, and HAL.

The automatic video algorithm measurements were compared against similar observationally measured properties. The motions observed in each video clip were timed using frame-by-frame analysis utilizing the Multimedia Video Task Analysis™ (MVTA) software tool. The analyst reviewed the videos and manually measured repetition frequency and duty cycle by annotating cyclic motions on a frame-by-frame basis.

A cycle in the laboratory task was defined by the elements (1) reach for a bottle, (2) grasp and move bottle, (3) release bottle and return to the starting position, and (4) rest (Figure 3). For this particular task cycle time was the time elapsed between each repetition cycle (i.e. start of Element 1, reach for a bottle). Hand exertion time was the time elapsed between grasping a load and releasing it (i.e. Element 2, grasp and move bottle).

Figure 3.

Selected video frames illustrating the task cycle: (1) reach for a bottle, Frame 207, (2) grasp and move bottle, Frame 240, (3) release bottle and return to the starting position, Frame 257, and (4) rest, Frame 293.

The HAL was estimated as a linear combination of the frequency of the movement time (1/f) and the duty cycle (D) according to the equation:

$$\mathrm{HAL}=4.31-0.637\times (1∕f)+0.034\times D.$$ (1)

If the equation returned a negative HAL it was considered zero. The equivalent HAL values for corresponding combinations of frequency and duty cycle according to the Hand Activity Level Threshold Limit Value (American Conference of Governmental Industrial Hygienists, 2009), are included with the HAL values from Equation 1 in Table 1. All 84 videos were processed using the automated video analysis framework and manually using MVTA to measure the actual frequency and duty cycle. To reflect the true algorithm performance, the HAL values were not rounded to the nearest integer.

Video Tracking Algorithm

An algorithm was programmed in C# with the open source OpenCVSharp (.Net wrapper for the OpenCV) vision library. The cross correlation based tracking algorithm was initialized by designating a rectangular region of interest (ROI) on the upper limb, in order to select the focal area on the hand or arm. The template matching tracking algorithm tracked the ROI motion trajectory over successive video frames. A cyclic motion analysis algorithm estimated the movement time based on zero-crossings of the ROI relative to a stationary gating region.

To elaborate, denote r_i to be a vector of intensity values of all pixels within the ROI at the i^th frame, and r_i+1(w) to be a candidate ROI at the (i+1)^th frame with a displacement of w ∈ Ω in a pre-specified search area. Ω is determined based on the prior knowledge of the speed of arm movement (pixels/frame). The cross-correlation between r_i and r_i+1(w) is defined as the angle between these two vectors (∣r_i∣ is the magnitude of the vector r_i, ${\mathbf{r}}_i^T$ is the transpose of r_i):

$$R\left(w\right)={\mathbf{r}}_i^T{\mathbf{r}}_{i+1}\left(w\right)∕(\mid \mid {\mathbf{r}}_i\mid \mid \cdot \mid \mid {\mathbf{r}}_{i+1}(w)\mid \mid )$$ (2)

The displacement w^* = arg. max._W R(w) determines the updated position of the ROI at the (i+1)^th frame, r_i+1 = r_i+1(w^*). The sequence of w^* is recorded as the ROI displacement between these two frames, and the new position of the ROI yields the motion trajectory at the (i+1)^th frame.

In this design, our approach for computing the cycle time was to monitor the crossing of the moving ROI and a stationary gating region in the video frame. A cycle was defined as the time elapsed between consecutive gating region crossings from the same direction. The recorded video data pixel location values x_vid, y_vid were low-pass filtered using a Butterworth filter. The velocity (v_x, v_y) and acceleration (a_x, a_y) vectors at each time stamp were calculated as follows:

$$\begin{array}{cc}\hfill & {\nu }_{p,i}=(p_{i+1}-p_{i-1})∕2\Delta \hfill \\ \hfill & a_{p,i}=(p_{i+1}-2\times p_i+p_{i-1})∕{\Delta }^2,\hfill \end{array}$$ (3)

where p∈ {x, y}, i ∈ {2, 3,…} and Δ = 1/30 second, which is the sample rate of the video. The video velocity and acceleration magnitude were determined by:

$$\begin{array}{cc}\hfill & {\nu }_{\mathit{xy},\mathit{video}}=\sqrt{{\left({\nu }_x\right)}^2+{\left({\nu }_y\right)}^2}\hfill \\ \hfill & a_{\mathit{xy},\mathit{video}}=\sqrt{{\left(a_x\right)}^2+{\left(a_y\right)}^2}.\hfill \end{array}$$ (4)

An algorithm was developed to automatically detect when the hand was loaded using velocity and acceleration of the ROI. It was observed that the peak ROI acceleration was greater when moving the load than when reaching and resting. This is illustrated by the histogram in Figure 4. A representative single-cycle velocity and acceleration curve from the video analysis is illustrated in Figure 5. The time period between two consecutive local minima points on the velocity curve was considered loaded if more than one points on the acceleration curve exceeded a threshold value. The hand was still considered loaded if more than one acceleration point exceeded threshold for the adjacent pair of local minima, (See Figure 5). The duty cycle D was then calculated as the average time when hand was loaded divided by cycle time. In our experiment, the threshold value was arbitrarily set by trial and error to the 85^th percentile acceleration when the tracked ROI was in motion (see Figure 5).

Figure 4.

Histograms for acceleration corresponding to the example in Figure 5. The dashed vertical line represents the acceleration threshold used as criterion for a loaded hand.

Figure 5.

Representative single-cycle velocity and acceleration plots for the laboratory load transfer task. The exertion time is the period in which the load is grasped, moved and released while the cycle time is represented by the task period determined by the video tracking algorithm.

To further illustrate the algorithm used to determine when the hand is loaded, refer to the example in Figure 3. The small red square represents the tracked ROI, which contains approximately 900 pixels (30 × 30). The stationary gating region was selected near the bottle dispenser. The local minima on the velocity curve are labeled 1-11 in Figure 5. The algorithm compares two consecutive local minima points sequentially, and determines when the hand was loaded using the criteria described above. From the example shown in Figure 5, the algorithm detected that the hand was not loaded between local minima pairs 1-2, 2-3, and 3-4; while the hand was loaded between local minima pairs 4-5 and 5-6.

RESULTS

The automatically predicted frequency, duty cycle and corresponding HAL was estimated for the duration of each video clip and is plotted against the corresponding manually measured MVTA values in Figure 6. A linear regression line was estimated with the intercept is set to 0. The slope of these three plots were 0.98 (R²=.79), 1.27 (R²=.63), and 1.06 (R²=.77) for frequency, duty cycle, and HAL respectively.

Figure 6.

Algorithm predicted result compared with MVTA manually measured result, where (a) frequency, (b) duty cycle, and (c) HAL are shown (N=84).

Summary statistics for predictions of HAL, frequency and duty cycle are provided in Table 2. Averages are among the 12 participants for each HAL condition.

TABLE 2.

Average (standard deviation) of the automated video analysis.

Paced HAL	HAL by eq. (1)	Predicted HAL	Frequency (Hz)	Duty Cycle (%)
1	0.1	1.2 (0.55)	0.17 (0.03)	23.97 (10.81)
2	2.1	2.1 (0.74)	0.25 (0.07)	15.30 (6.11)
2	2.4	2.9 (0.63)	0.28 (0.07)	29.48 (10.50)
4	3.7	3.2 (0.36)	0.30 (0.04)	31.58 (5.47)
4	4.1	4.1 (0.20)	0.37 (0.07)	45.06 (9.15)
5	4.7	5.0 (0.44)	0.43 (0.09)	64.80 (9.96)
5	5.4	5.4 (0.41)	0.76 (0.10)	56.74 (11.32)

DISCUSSION

For the two paced HAL = 2 conditions (0.25 Hz from Table 1), the difference between the average observed frequency and the paced frequency ranged from 0 Hz and 0.03 Hz, respectively. For a paced HAL = 3, 4 and 5 (0.5 Hz from Table 1), the differences between the average observed frequency and the paced frequency ranged from −0.2 Hz to −0.07 Hz. When HAL = 5 (1 Hz from Table 1), the difference was −0.24 Hz.

Although the aforementioned comparison seemed reasonable as a group, it does fully not account individual variability. During the experiment, we found that each person performed somewhat differently to the audio cues, despite the fact that they are given the same oral instructions and same amount of practice period. Individuals also responded differently when dealing with fast and slow pacing.

The average predicted HAL was closest to the paced HAL when HAL exceeded 1. In the case of paced HAL = 1, the duty cycle was under-estimated and the frequency was over-estimated resulting in higher predictions (Figure 4). This was mainly due to the selection of the velocity magnitude threshold. For low HAL conditions, movement speed within the video frame was small. The slow movement speed was filtered out by the predefined threshold. This lowered the estimation of active time (hand motion time), and hence resulted in higher estimates of frequency.

Occlusion and disappearance of the ROI also pose challenges. When a hand is occluded by a foreign object, the video tracking may fail and the region being tracked may be high jacked by the foreign object. Self-occlusion is also possible, when the person turns against the camera and therefore the hand region may be occluded by the body. Disappearance of the hand and arm is also possible, since the person shooting the video may underestimate the job activity region. When the hands reach beyond the field-of-view of a camera, there is a great chance that the current tracking system will fail. Camera motion causes undesired video input, such as a blurry image and temporary disappearance of the ROI.

In practice, selection of a ROI that minimizes disappearance is highly dependent on the specific task, work layout, camera location, and other workplace conditions. Our methodology requires the analyst to carefully identify the specific location of the ROI on the body, whether it is hand, forearm, arm or shoulder, by pinpointing a body part that is directly related to the hand-arm motion and is visible to the camera. In many cases under actual workplace conditions, these measures by themselves may be insufficient to fully eliminate disappearance. The camera angle and location are additional considerations. Future research will study the magnitude of these limitations and strategies for minimizing their effects.

To address these problems through image processing, it is crucial that the tracking system make use of the past trajectory history. By incorporating a suitable statistical state transition model, such as the Kalman filter, we may be able to predict the new ROI position. This predicted new position will be used as the reference position for next tracking cycle. Every iteration the state model location is updated and corrected in order to maintain the accuracy of the prediction.

One way to compensate for camera motion is to track background patches or strong corner points, in addition to tracking the ROI. The motion of these patches or corners can be averaged together and provide an estimate of the camera motion (Adams, et al. 2008). Another way to estimate camera motion is obtaining help from devices such as gyroscope or accelerometer, which provides the orientation and motion information of the camera. The estimated camera motion information can later be used to correct the hand displacement quantity and provide better estimates of the HAL number.

Different camera locations and angles can impose further limitations on the current method. One most obvious constraint is when viewing the subject from one side, the contralateral side is occluded by the body. Currently, our method has been tested in the laboratory for a task that was limited in field-of-view changes. We are investigating how the threshold selection affects the performance of the HAL estimates, and whether additional variables, such as patch size, should be incorporated in the threshold selection process.

The measured velocity and acceleration in the two-dimensional video may be affected by subject movements outside of the plane. Different camera angles may also affect algorithm performance, including parallax, and this issue will be assessed and addressed in future studies. Future research will consider repetitive tasks that vary in body movements, directions and exertions. The effect of ROI selection on HAL estimates using various limb segments is another area of future research. Future improvements to the algorithm will involve a systematic method for differentiating between the active and inactive portion of a cycle. Currently the threshold values were determined by experience and trial and error and were specific to the laboratory repetitive motion task performed in this study.

The ACGIH TLV uses measurements of normalized peak force in addition to HAL for assessing exposure limits. HAL is strictly related to frequency of motion and duty cycle. Exertions, whether static or dynamic, are independent of HAL and are considered separately in the TLV by measuring exertion levels. This methodology is limited to coordinated motions that are observed in the video record and hand exertions that may not visible are not measured. The algorithm and the results of this study were limited to one particular load transfer task described in this proof-of-concept study. This provides a starting point by which similar algorithms can be developed for other tasks. The algorithm has not yet been tested in alternative tasks such as using a computer keyboard or jobs that involve prolonged static postures. The algorithm currently does not measure static exertions where forces are made without movements. For example, if an individual gains control of an object and then continues to grip it without motion, it would extend the work time, the cycle time and the duty cycle. Future research should address exertions and account for these types of activities.

For the task in this study, the peak ROI acceleration was greater when moving the load than when reaching and resting. This observation was used to segment the cycle. It is likely that additional criteria might be needed (other than peak acceleration) for different tasks. Algorithms for detecting exertions based on ROI kinematics needs to be further evaluated and developed and will be the subject of future research. In the current experiment, the threshold value was arbitrarily set to the 85^th percentile acceleration when the tracked ROI was in motion. A general acceleration threshold algorithm should be developed for duty cycle detection. The method used in the current study sufficed for proof of concept purposes.

We conclude that the video-based direct exposure assessment method shows promise. The use of marker-less video is unobtrusive and does not require attaching sensors to the body of workers, which often interferes with the job and possibly movement patterns and exertions. Furthermore such an application might be ported to programmable, camera-enabled mobile devices. This could lower the instrumentation barrier and make routine analysis of upper limb work-related occupational hazards more accessible to general industry.

Key Points.

• The new marker-less video algorithm emphasizes temporal patterns, rather than the high precision normally important for spatiotemporal motion tracking, and employs a semi-automatic approach whereby the human analyst interactively selects a region of interest associated with the repetitive task to track, rather than imposing an a priori model of the tracked activity.

• The approach is objective, unobtrusive and does not require attaching sensors to the body of workers, and suitable for a real-time, direct reading exposure assessment instrument for hand activity level.

• The video-based direct exposure assessment method shows promise for a simple hand load transfer task in this proof-of-concept study and provides a starting point by which similar algorithms can be developed for other tasks.

BIOGRAPHIES

Chia-Hsiung Chen is a PhD student in Electrical and Computer Engineering at the University of Wisconsin (Madison, WI) and a research assistant in the Occupational Ergonomics and Biomechanics Laboratory. He earned his BS degree from the National Sun Yat-Sen University (Taiwan), and MS degree from the National Taiwan University (Taiwan).

Hu Hen Hu is a professor at the University of Wisconsin (Madison, WI) in Electrical and Computer Engineering. He has a BS degree (1976) from the National Taiwan University (Taiwan), and an MS degree (1980) and PhD (1982) from the University of Southern California (Los Angeles, CA).

Thomas Y. Yen is an associate instrumentation innovator and instructor at the University of Wisconsin (Madison, WI) in the departments of Biomedical Engineering and Industrial and Systems Engineering. He has a BS degree (1987) from Northwestern University (Evanston, IL) and an MS (1991) and PhD (1997) from University of Wisconsin (Madison, WI).

Robert G. Radwin is a professor at the University of Wisconsin (Madison, WI) in Biomedical Engineering, Industrial and Systems Engineering, and Orthopedics and Rehabilitation. He has a BS degree (1975) from New York University Polytechnic Institute, and he earned MS degrees (1979) and a PhD (1986) from the University of Michigan (Ann Arbor, MI).