Classifying hazardous movements and loads during manual materials handling using accelerometers and instrumented insoles

Abstract

Improper manual material handling (MMH) techniques are shown to lead to low back pain, the most common work-related musculoskeletal disorder. Due to the complex nature and variability of MMH and obtrusiveness and subjectiveness of existing hazard analysis methods, providing systematic, continuous, and automated risk assessment is challenging. We present a machine learning algorithm to detect and classify MMH tasks using minimally-intrusive instrumented insoles and chest-mounted accelerometers. Six participants performed standing, walking, lifting/lowering, carrying, side-to-side load transferring (i.e., 5.7 kg and 12.5 kg), and pushing/pulling. Lifting and carrying loads as well as hazardous behaviors (i.e., stooping, overextending and jerky lifting) were detected with 85.3%/81.5% average accuracies with/without chest accelerometer. The proposed system allows for continuous exposure assessment during MMH and provides objective data for use with analytical risk assessment models that can be used to increase workplace safety through exposure estimation.

1. Introduction

The most common class of non-fatal work-related injuries regardless of industry is from overexertion. Many of these injuries are the result of manual materials handling (MMH) (Bhattacharya and Pandalai, 2015). Work related musculoskeletal disorders (WMSDs) accounted for 30% of all days away from work (DAFW) cases in 2018 for a total of 272,780 cases (U.S. Bureau of Labor Statistics, 2020). In the same year, WMSD cases in professions specifically involving MMH activities (i.e., construction, freight, and stock) and patient handling activities (PHA) (i.e., nursing and healthcare) represented 40-50% of all DAFW cases in their respective fields (U.S. Bureau of Labor Statistics, 2020). WMSDs often lead to chronic conditions or disabilities, incurring many additional direct and indirect costs totalling $13.3 billion USD in the 2021 Liberty Mutual workplace safety index (Liberty Mutual Insurance, 2021). Within the umbrella of WMSDs, lower back injuries and lower back pain (LBP) represent 51% of all WMSD cases (Umer et al., 2018), representing injuries with the highest cost and health burden (Lawrence et al., 1998). Improper lifting techniques; pushing, pulling, lifting, and carrying heavy loads; twisting and bending of the back; frequent repetition of physical tasks; prolonged standing and sitting; and whole-body vibrations have been shown to lead to WMSDs (Marras et al., 2010; Parakkat et al., 2007; Murtezani et al., 2011).

To characterize exposure during MMH tasks and mitigate WMSD and injury risk, several ergonomic assessment tools and methods have been developed. The existing risk assessment methods such as the Revised NIOSH Lifting equation (RNLE) (Waters et al., 1993, 1994; Lu et al., 2014), the Assessment Tool for Repetitive Tasks (ART) (Ferreira et al., 2009), and Manual Handling Assessment Charts (MAC) (Monnington et al., 2002; Pinder, 2002) relate factors such as posture, frequency, load, and duration of the repetitive movements to assess injury risk. Many of these factors can be obtained using observational methods (Spector et al., 2014) or direct measurements (e.g. using wearable sensors, accelerometers, or inertial measurement units (IMUs)) (David, 2005; Donisi et al., 2021). Unfortunately, IMU measurements cannot directly estimate the loads that are lifted or carried, which is one of the most important parameters in risk assessment analytical tools. Additional limitations with these risk estimation methods include: reliance on cross-sectional or case-control studies, dependence on job titles or self-reported physical exposure resulting in imprecise exposure estimates (Kilbom, 1994), and inadequate techniques to quantify complex physical exposures for tasks with varying loads, hand locations, and/or job rotation (Antwi-Afari et al., 2019). Moreover, the multiple complex tasks performed by a single worker in modern industries requires continuous monitoring for accurate exposure assessment.

Advances in sensing and analytical methods (Marras et al., 1992; Feyen et al., 1999; Chaffin, 2007; Vignais et al., 2013) along with recent developments in wearable technologies have laid the groundwork for longitudinal and advanced monitoring of workers’ exposures (Faber et al., 2016). Better exposure data from wearable technologies and smart algorithms can enhance our ability to predict chronic incidences of WMSDs and their implications on total worker health.

Determining work activities and task timings using wearable sensors for ergonomic risk estimation methods are feasible means of exposure estimation, yet has proven difficult (Kim and Nussbaum, 2014; Maman et al., 2017; Lin et al., 2016). Combining a machine learning approach with body-mounted IMU measurements has been shown to achieve 90% verification accuracy in three task classifications (Nath et al., 2018). Applying the same methods to differentiating individually between correct and incorrect lifting posture showed verification accuracies of up to 99.4% with 8 IMUs and 76.9% with a single trunk IMU (Conforti et al., 2020). A single accelerometer on the chest was also employed to classify 15 different simulated activities with 93%-98% accuracy (Hosseinian et al., 2019). Recently, it was shown that a single IMU sensor can be used to detect MMH activities with 78%-86% accuracy, with its specific location affecting the accuracy of the detected activities, while multi-sensor configuration can achieve accuracies of >97% (Porta et al., 2021). However, relying on measurements from only IMUs does not provide information about the load, which has been shown as an important independent risk factor (David, 2005).

The use of only instrumented insoles to detect and classify MMH events may overcome many of these limitations. It is a promising alternative solution, due to its minimally invasive measurement approach that does not require workers to wear any body-mounted sensors, and is appropriate for use in various workplaces including harsh working environments that require wearing other personal protection equipment (PPE). Kim and Nussbaum (Kim and Nussbaum, 2014) included information from pressure sensitive insoles in addition to 17 wearable IMUs and classified six generic MMH tasks with 90% accuracy. The classification algorithm had difficulty identifying the start and end time of the task and no load information was provided (Kim and Nussbaum, 2014). Wearable and Connected Gait Analytics System (WCGAS) developed in (Chen et al., 2018) used the plantar pressure distribution map images and Kinect skeleton model to classify the MMH activities. Only quasi-static postures were analyzed without detecting the actual values of force exertions. Antwi-Afari et al. (2018) demonstrated a high overall verification accuracy (99.7%) in distinguishing between five static MMH tasks using only plantar pressure distribution and a SVM classifier. Similarly, only static postures were considered with no dynamic tasks or task transitions, and tasks were classified individually. In a later study from the same group (Antwi-Afari et al., 2019), a combination of static and dynamic tasks were included reaching classification accuracy of 94%, although no task transitions and lifting load characterization was reported. In addition, load characterization using footwear sensors and a pressure-map manifold was previously investigated during patient handling activities (Lin et al., 2017). However, only a qualitative load description was provided (i.e., no, low, or high) and no actual values were measured or classified. While the instrumented insoles have shown promise in detecting the qualitative loads, the largest shortcoming when entirely removing IMUs from the system is the lack of sufficient data for distinction between activities where the wearer’s weight is not supported by the feet (e.g. double kneeling or crawling) (Antwi-Afari et al., 2019).

The goal of this study was to develop an algorithm to detect and classify MMH activities as well as the load lifted and carried based on the measurements from instrumented insoles and a single chest-mounted accelerometer. The algorithm would have the capability to extract important exposure parameters to be used for injury risk estimation of workers performing MMH tasks, including task duration, cumulative activity distributions, and lifting frequency.

2. Methods

2.1. Participants

Six young adult males; mean ± SD; age: 29.7 ± 3.2 years; height: 179.2 ± 4.1 cm; mass: 78.5 ± 8.9 kg; shoe size: 27.6 ± 0.6 cm; with no prior musculoskeletal disorders or back pain history participated in the study. We restricted the inclusion of participants by requiring them to be capable of and comfortable lifting a 15 kg load. Prior to the test, the participants were informed about the study protocol and signed the informed consent form approved by the Institutional Review Board at the University of Utah.

2.2. Experimental protocol and tasks

The participants performed MMH tasks as shown in Fig. 1. These tasks included lifting/lowering, side-to-side transferring, carrying a box, standing and walking without carrying a load, as well as pushing or pulling a cart. The mass of the box and lifting behavior varied during the test. The performed tasks are described below:

Figure 1:

Manual material handling tasks performed by the participants during experimental testing. (a) Lifting in a squat (left) and stoop (right) postures. (b) Overextended lifting (left) and lifting with jerky behavior (right). (c) Asymmetric lifting with twisting torso and stationary feet (left) and carrying the box with proper positioning in front of the load (right). (d) Lifting/lowering of the box from the table/knuckle height to the chest height from right to left side (left) and left to right side (right). (e) Pushing (left) and pulling (right) a cart. (f) Lifting a box from the ground (left) and from the table (right) and carrying it.

1. ‘LiftLowerSquat’ and ‘LiftLowerStoop’: Participants used both hands to lift the box from the floor on to the table and lower it from the table on to the floor. In between each lifting/lowering, the participants paused for 2 seconds. Tests were repeated for 5.7 kg and 12.5 kg loads, and for a squat and stoop posture; see Fig. 1a. During both tasks, the horizontal distance of the handholds on the box from the participant’s center line was approximately 40cm.

2. ‘LiftExtend’ and ‘LiftJerky’: Participants performed the overextended lifting of 5.7 and 12.5 kg loads positioned 63 cm in front of the participant (Waters et al., 1994) on a table and jerky lifting from the floor. The participants reached over with both hands and moved the box on the ground. After a 2 sec rest, the participant lifted the box in a fast jerky manner and positioned it on the table; see Fig. 1b.

3. ‘SideAsym’: Participants asymmetrically lifted the box from one side to another (90 degrees in either direction) using body rotation and minimal movement of feet. Distance between participants’ mid plane and each table was 63 cm; see Fig. 1c (left).

4. ‘SideShoulder’: Participants lifted/lowered 5.7 and 12.5 kg box from the table/knuckle height on to the cabinet of chest height, turning slightly; see Fig. 1d. Participants stood and rested for 2 sec between each lift.

5. ‘Carry’: Participants lifted and carried the box for 10 m then lowered the box down. Two different box weights (5.7 and 12.5 kg) and initial/end positions (lifting from the floor and lifting from the table height) were considered; see Fig. 1f. In addition, participants carried box for a short distance from one table to another without any trunk asymmetry, and position their feet and themselves in front of the table to mimic short carrying tasks; see Fig. 1c (right).

6. ‘Push’ and ‘Puli’: Participants pushed the cart with both hands for 5 m and paused for 2 sec. Then pulled the cart back to the initial position; see Fig. 1e. The load on the cart was 85 kg.

7. ‘Stand’ and ‘Walk’: Rest activities during the above-mentioned activities were classified as standing. Participants walked the same path and distance as during carrying.

Activities were grouped into four test sets. Each set lasted approximately 20 minutes. There was a 5 min-utes rest period between each test sets and data was not recorded during that time. Test sets were assigned to subjects in random order and only performed once per subject to prevent learning effects. Test set 1 included lift/lower tasks (Tasks A and B) between the floor and table, each performed repeatedly for approximately 3 minutes. Test set 2 included lifting tasks SideAsym and SideShoulder (Tasks C and D) between two tables of same or increasing heights and on both sides of the table/cabinet, with each task performed repeatedly for approximately 3 minutes. Test set 3 included only carry tasks (Task E). Test set 4 included simple locomotion tasks Pull and Push (Task F) at a natural pace, followed by walking the same distance (Task G).

The same box with different weights inside was used in all lifting experiments. Box size was 33 × 33 × 28 cm and had proper handles positioned 28 cm from the lower edge. Table height was set at 73 cm for all participants, approximately at the knuckle height while standing. The cart handle height was 90 cm from the floor. The height of the cabinet was 115 cm.

2.3. Risk Assessment

Six lifting activities (Stoop, Squat, Overextended, Jerky, Side Asymmetrical, and Side Shoulder) were analyzed using the RNLE (Waters et al., 1994) to demonstrate the ability of the machine learning classifier. Frequency Independent Recommended Weight Limit (FIRWL) was calculated initially, then used to calculate Recommended Weight Limit (RWL) and Lifting Index (LI) using classifier outputs for activity, load, and frequency. Table 1 shows FIRWL values for the given six lifts, holding frequency and coupling multipliers constant at a value of 1. FIRWL was computed at the start and end of each lifting activity, with the minimum FIRWL considered as the recommended limit for the entire activity. Five out of the six activities showed similar limits for the start and end of the task, with less than 1 kg difference between the two. One activity (overextended lifting) showed a significantly lower limit at the table end of the lift, due to overextension when moving the load.

Table 1:

Frequency Independent Recommended Weight Limit (FIRWL) for lifting activities.

	FIRWL (kg)
Activity	Start	End
Stoop	10.9	11.6
Squat	10.9	11.6
Extend	10.9	7.4
Jerky	10.9	11.6
SideAsym.	6.5	6.5
SideShoulder	7.6	6.7

Loads were chosen based on the maximum (11.6 kg) and minimum (6.5 kg) FIRWL in Table 1. A weight guaranteed to be safe for all activities was chosen at approximately 10% below the minimum recommended load, while a weight guaranteed to be risk-probable for all activities was chosen at approximately 10% above the maximum recommended load. Thus, loads of 5.7 and 12.5 kg were used for “safe” and “risky” lifting respectively. The exact weight steps were limited by the combinations of weights available for the study, forcing values slightly larger or smaller than the target 10%. Frequency Independent Lifting Index (FILI) values with these two loads come to a maximum of 0.88 for the safe load during most risk-prone lift, and a minimum of 1.08 for the risky load during the safest lift.

Pushing and pulling activities are not covered by the RNLE. Push and pull were instead analyzed using the Risk Assessment for Pushing and Pulling (RAPP) (Health and Safety Executive, 2016). RAPP defines the cart used in this study (self-supported movable structure on 3+ castor wheels without a steering mechanism) as a medium size cart. For medium sized carts, a safe or “low” level of risk is defined as < 250kg. Given the predefined static load (85 kg, “low”) and travel distance (5m, “short”) with optimal environmental variables (posture, “reasonable”; hand grip, “good”; equipment condition, “good”; floor surface, “good”; obstacles, “none”), the greatest potential for variation in risk level came from the frequency of the task.

RAPP provides a risk analysis, but does not differentiate between push and pull tasks of the same conditions. However, pulling can exert higher forces on the body, particularly the lower back (Pinupong et al., 2020), making pushing the recommended action over pulling when given the same load and environmental variables (Liberty Mutual Insurance, 2017). Thus, it is important that they are detected separately.

2.4. Measuring systems and data collection

The participants wore instrumented insoles (Science Pro+, zFLO Motion, Westbrook, ME; produced by Moticon GmbH, Germany) (Braun et al., 2015; Stöggl and Martiner, 2017; Oerbekke et al., 2017) and an accelerometer attached to the chest (Delsys Trigno wireless accelerometer, Delsys Inc., Natick, MA). Each insole contains 13 pressure sensitive cells, an integrated 3-axis accelerometer, and on-board memory for saving recorded data. The center of pressure (COP) in a shoe relative coordinate frame (x- and y-axis COP) are calculated and provided as the output signals from each insole. Prior to the start of the test of each participant, all force/pressure readings were zeroed to reduce the effect of pressure due to tightening of the shoes. Insole data were collected at 50 Hz sampling frequency and data from the accelerometer was recorded at 1926 Hz and downsampled to 50 Hz. Simultaneously, we recorded and synchronized video of participants performing MMH tasks. The activities were labeled by the same observer for all participants to minimize the labeling discrepancies. The start and end of the event were determined as the instant when a secure grip on the box or cart was established and there was a noticeable exertion applied to move the load. The end of the event was determined as the instant when the box was securely placed or when the cart came to rest and the participant released the grip from the box or cart.

2.5. Dependent variables

In our analysis, we considered sixteen dependent variables. We summed the force contributions from all 13 sensors separately for the left and right insole to obtain variables, f_zL and f_zR, respectively. These signals represent the normal ground reaction forces per individual foot. Both signals were summed to obtain the resultant normal ground reaction force F_z. The location of the COP along the x- and y-axis in the shoe relative coordinate systems were considered for each insole, representing variables COP_xL, COP_yL, COP_xR, and COP_yR. To accommodate shoe size variations among the participants, the COP data were normalized as COPnorm_ij= COP_ij /ShoeSize_k, i = x, y, j = L, R, where ShoeSize_k is the shoe size of the k-th participant and k = 1, …, N, where N is the total number of all participants.

Three accelerations from each insole in their corresponding relative coordinate frames, ACC_ij, i = x, y, z, and j = L, R, were included as the individual variables. An additional three acceleration measurements ChestAcc_ij, i = x,y, z, j = L, R, were included from an accelerometer mounted on the participants’ chest. In total, three force measurements, four COP signals and nine accelerations were considered in this analysis.

2.6. Data processing

To partially suppress the effects of body segment dynamics, we low-pass filtered the kinetic (i.e., forces/pressure) signals using a second-order Butterworth filter with a 1 Hz cut-off frequency (Trkov and Merryweather, 2018). Kinematic (i.e., accelerations) and COP data from the insoles were filtered using a fourth-order Butterworth filter with a 15 Hz cut-off frequency to eliminate sensor noise. Yu suggests common cutoff values of 3-15 Hz for acceleration filtering (Yu et al., 1999), due to the additional step of feature extraction being performed, the higher value of the range was used to preserve these features.

After filtering, data were pre-processed and prepared to be input into previously selected machine learning classifiers. Each frame of data included a label of activity and lifting/carrying load used (i.e. no load, 5.7 kg, or 12.5 kg). Feature extraction was performed using a moving buffer. The initial buffer size of 2 seconds was chosen based on a previous study (Hosseinian et al., 2019), however, a larger range (0.5-3 seconds with an increment of 0.5 seconds) was tested to find the optimal buffer size that resulted in the highest overall accuracy. The sampling rate with which the buffer processed data was 10 Hz, chosen for 95% overlap between buffers. Extracting data in excess of 95% overlap only created redundant data points which did not significantly improve classifier accuracies.

For data contained in each buffer, six statistical features were computed: average, standard deviation, maximum, minimum, range, and kurtosis (Antwi-Afari et al., 2018). A total of 96 signal features were calculated from 16 input signal variables.

10% of the total data available were separated at random to be used as an independent test set. The remaining 90% of the data were randomized and used to train classifiers with 5-fold cross validation. Due to the nature of the data collection, some actions inherently had more data than the others. Standing still with no load ‘Stand’ in particular made up ~50% of all data. This was remedied by under-sampling standing data at half the sampling rate to make it approximately equal to the next most prevalent activity (Carry, 5 & 12 kg). Training set data distribution is shown in Figure 2. This was combined with cross-validation to decrease overfitting on activities with more data. Under-sampling was not performed on the test set to better simulate the type of data that the classifier would be given to process.

Figure 2:

Distribution of data per activity type and weight.

2.7. Machine learning classifiers

The classification ability of the models was evaluated primarily on the overall accuracy considering MMH events with load distinction. Accuracy was calculated as shown in Eq. 1. As a secondary measurement, a high sensitivity or True Positive Rate (TPR) was considered of more importance than a high specificity or True Negative Rate (TNR), calculated with Eq. 2 and Eq. 3 respectively. False Negative Rate (FNR) as shown in confusion matrices is computed as the distance of TNR from 1. These measures were chosen because it is more important for the classifier to over-predict than under-predict exposure. While over-predictions could lead to false alarms, under-predictions can lead to undetected injuries. All prediction rates were calculated individually for each class without considering other classes, then the normalized values were averaged.

$$Accuracy=\frac{TruePositives+TrueNegatives}{TotalSamples}$$ (1)

$$TPR=\frac{TruePositives}{TruePositives+FalseNegatives}$$ (2)

$$TNR=1-\frac{TruePositives}{TruePositives+FalseNegatives}$$ (3)

Five supervised machine learning classifiers were chosen to evaluate classification accuracy. The classifiers included Single Binary Decision Tree (ST), Bagged Trees Ensemble (BT), k-Nearest Neighbors (KNN), Support Vector Machine (SVM), and Naive Bayes (NB) that have previously shown good performance in similar applications (Antwi-Afari et al., 2018; Anjum and Ilyas, 2013).

Single Binary Decision Tree (ST).

The single tree classifier is a simple classifier, chosen for its extremely light computational load and previous good performance in related studies (Antwi-Afari et al., 2018, 2019). Analyzing the differences between samples creates sets of interconnected tests called nodes. New examples are classified by creating a path through these nodes, with the final node being the classification label (Aly, 2005).

Bagged Trees Ensemble (BT).

Using a variable number of decision trees in a bagged tree classifier allows for increased accuracy over the single binary decision tree using the same principles.

Naive Bayes (NB).

The Naive Bayes classifier separates activities by maximizing aposterior probability, calculated using Bayes’ theorem (Aly, 2005). It was chosen based on previous successful detection of human daily physical activities using wearable sensors (Anjum and Ilyas, 2013).

k-Nearest Neighbors (KNN).

The KNN is a standard machine learning algorithm, frequently used in occupational activities classification studies (Antwi-Afari et al., 2018, 2019; Lin et al., 2017; Kim and Nussbaum, 2014). It uses a similarity function, and classifies new examples the same as their closest neighbors in terms of the given function (Aly, 2005). The similarity function used here is euclidean distance.

Support Vector Machine (SVM).

Support vector machines are typically a binary classifier that can be extended to multi-class problems by projecting the data onto a higher dimension space (Aly, 2005). The SVM is considered to be a more intuitive and robust classifier than those previously discussed, based on minimizing the distance from a respective class-separating hyperplane in a higher dimension (Aly, 2005). It has been shown to outperform other classifiers in similar activity detection using wearable insole data (Antwi-Afari et al., 2018).

2.8. Continuous lifting frequency

Automating the accurate collection of the frequency and duration for which activities are performed will allow for more accurate assessment of workers’ exposures over longer periods. The lift frequency over time for each activity was computed based on the detected activities obtained from the classification process. Raw predictive data was smoothed before calculating frequencies. Activities detected for less than 0.5 seconds continuously were treated as noise, and filtered out. With the noise removed, lifting frequency was computed over continuous time with a sliding window of one minute by summing the total number of occurrences within the window for each point in time. Frequency derived from model prediction was compared to frequency derived from manually labelled activities, computed in the same way. To maintain proper time series presentation, frequency estimation was only performed on a continuous test set.

3. Results

We compared the performance of the following five classifiers: K-Nearest Neighbors (KNN), Bagged Trees (BT) Ensemble, Support Vector Machine (SVM), Single Binary Tree (ST), and Naive Bayes (NB), using a test data set with 2 sec time window and 95% overlap. The top three performing classifiers for test data considering load distinction were SVM, KNN, and Bagged Trees with average overall accuracies of 85.3%, 85.3%, and 82.9%, respectively.

Figure 3 shows average accuracies for all classifiers with and without detecting differences in lifting and carrying loads. During this initial testing, Single Tree (67.9% accuracy) and Naive Bayes (69.2% accuracy) classifiers did not perform well and were not considered in further analyses. Considering only activities without distinguishing lifting load was found to offer very little additional predictive power to the model overall, as accuracies on best performing models increased by less than 2%. Therefore, classifiers with weight distinctive data were used due to the negligible cost of the added utility.

Figure 3:

Results of classification accuracies for all classifiers (i.e., SVM, KNN, Bagged Tree, Naive Bayes, and Single Tree) with and without considering load distinction.

The top two performing classifiers were selected for further analysis. We performed buffer size analysis and parameter optimization. The KNN classifier was trained using euclidean equal distance weights and the tested k-value varied between 1 and 10 in steps of 1. Using k=5 performed best with an overall accuracy of 85.3%, an average sensitivity of 80.1%, and an average specificity of 76.7%. This k-value is similar to those used in related classification studies (Lin et al., 2017). The SVM used a gaussian distribution with a kernel scale varying from 1-10. The kernel scale of 7 performed best with an overall accuracy of 85.3%, an average sensitivity of 88.3%, and an average specificity of 78.7%. Figure 4 shows that the optimal window/buffer size was 2 seconds that resulted in the highest overall accuracy for both SVM and KNN classifiers. The optimal window size of 2 seconds was used in all classifications. Optimal KNN and SVM accuracies were equal, however, the SVM classifier showed higher values of both sensitivity and specificity, and was used for the remainder of the study.

Figure 4:

Effect of the window size (i.e., buffer size) on the overall classification accuracy of SVM and KNN classifiers for data considering load distinction.

We performed the parameter analysis of data obtained from different sensors to investigate their effects on the overall accuracy of the classifier. Four different types of data (ground reaction forces (GRF), COP coordinates, and accelerations (ACC) from each shoe, and chest-mounted accelerometer (Chest Acc)) were considered individually and in combinations. Table 2 shows the comparison of accuracy results on independent test data for nine sensor combinations that were analyzed for the top performing (SVM) classifier. Classification when considering all parameters performed best with accuracies of 85.3%. Classification results using only measurements from the instrumented shoe sensors resulted in significantly lower accuracies of 80.7%. The inclusion of GRF data was shown to produce the highest accuracies when used in combination with ACC or COP, likely because it is the best separator of load weights. However, ACC signals from either the foot or chest outperform both GRF and COP signals by themselves, implying that accelerations are the single best predictor of different activity types. Classification accuracies when using COP signals were always lower than counterparts using the same number of signals, suggesting that it is the least important signal to consider for activity classification.

Table 2:

Average accuracy presented for parameter sensitivity analysis on performance of the SVM classifier considering individual or combination of parameters measured by the instrumented insoles (i.e., ACC, GRF, and COP) and chest-mounted accelerometer (i.e., Chest Acc).

Predictors	Acc.
All (Shoe+Chest)	85%
All (Shoe)	80%
ACC/GRF	77%
GRF/COP	76%
COP/ACC	74%
ACC	68%
Chest Acc.	66%
GRF	64%
COP	63%

Table 3 shows the results of the accuracy, sensitivity, and specificity for each individual activity with and without load distinction for the trained SVM classifier. These results were calculated from the independent test dataset not used in validation. The highest consistent rates of prediction (accuracy and sensitivity both >90%) were carrying of both weights, pull, push, and walk. These activities all show more movement than any other activity while being performed. Thus, the classifier is best at predicting unique and dynamic tasks. Additionally, the only load-distinctive task in this group (carrying) showed similarly high values for both load levels.

Table 3:

Individual activity performance on a held out independent test set for SVM classifier with and without weight distinction.

Activity	w/ Weights			w/o Weights
	Sens.	Acc.	Spec.	Sens.	Acc.	Spec.
Stand_NoLoad	77.5%	84.5%	92.9%	75.9%	84.3%	94.7%
LiftLowerStoop_5.7kg	91.3%	82.1%	74.7%	96.3%	81.3%	70.3%
LiftLowerSquat_5.7kg	90.7%	70.8%	58.1%	93.2%	76.7%	65.2%
LiftExtend_5.7kg	94.3%	77.0%	65.0%	93.1%	77.0%	65.7%
LiftJerky_5.7kg	85.7%	92.3%	100.0%	100.0%	85.8%	75.1%
SideAsym_5.7kg	90.8%	84.1%	78.4%	93.8%	85.3%	78.2%
Carry_5.7kg	90.9%	91.0%	91.1%	95.9%	94.1%	92.3%
Pull_n/a	98.2%	95.0%	92.1%	97.9%	92.2%	87.1%
Push_n/a	92.1%	92.2%	92.3%	95.1%	95.4%	95.7%
SideShoulder_5.7kg	80.2%	77.2%	74.3%	83.6%	75.5%	68.9%
SideShoulder_12.5kg	92.5%	83.2%	75.6%	-	-	-
LiftLowerStoop_12.5kg	94.3%	82.9%	73.9%	-	-	-
LiftLowerSquat_12.5kg	88.0%	77.2%	68.7%	-	-	-
LiftExtend_12.5kg	97.0%	86.3%	77.6%	-	-	-
LiftJerky_12.5kg	86.6%	80.2%	74.7%	-	-	-
SideAsym_12.5kg	92.1%	85.0%	79.0%	-	-	-
Carry_12.5kg	92.4%	92.2%	91.9%	-	-	-
Walk_NoLoad	97.8%	93.3%	89.2%	97.6%	90.7%	84.7%

On average, all activities have a sensitivity equal to or greater than their specificity, meaning the frames where a given activity was predicted were greater than the times that other activities were mis-predicted as the given activity. However, four outliers include Stand, LiftJerky (5.7kg), Carry (5.7kg), and Push. These activities were often predicted incorrectly. Stand was treated as the default state for all tasks, so a lower specificity is expected due to periods of uncertainty at the beginning and end of activities, attributed either to uncertainty or error during manual labelling. Push and carry activities are also justified by their already high values for accuracy and sensitivity. The discrepancy for the LiftJerky (5.7kg) activity, however, shows that it is frequently (14.3% of the time) interpreted as the wrong task but never classified when another activity should have been. Thus, the issue is likely twofold - decrease in activity due to speed of performance, and increase in uncertainty due to the length of the activity in relation to the length of the sliding window. Lifting activities overall averaged 81.5% accuracy, with activities LiftLowerSquat (5.7kg), LiftExtend (5.7kg), SideShoulder (5.7kg), LiftLowerSquat (12.5kg), and LiftJerky (12.5kg) less accurate than the mean. These activities are also classified with <80% accuracy when lifting load is not distinguished, suggesting that these activities themselves are more difficult to distinguish between in addition to the load level.

Figure 5 shows the confusion matrix for the SVM classifier applied to the independent test set, with TPR and FNR given in cells for each combination of true and predicted activities. This chart gives further insight into the less accurate activities. Both load levels of LiftLowerSquat show similar amounts of mis-classification between load levels, which make up the most improper predictions for the two classes, confirming that load level is the most difficult factor to differentiate. This behaviour is replicated in LiftJerky and LiftExtend activities, but not in LiftLowerStoop which is primarily mis-classified as standing. Activities performed while standing still (SideShoulder, SideAsym) similarly show most of their mis-classifications as standing, with the most standing mis-classifications from SideShoulder activities, classified as standing 21.2% of the time. These activities show comparatively few mis-classifications between load levels, however, suggesting that the activities themselves are more difficult to distinguish than the load level in this case.

Figure 5:

Confusion matrix results for SVM classifier on an independent test set. Cell values show row normalized values (TPR, FNR).

Figure 6 shows the actual (solid) and computed (dotted) lifting frequencies over time for activities performed in test set 1. Noise was filtered from the predicted activities by disregarding activity signals that were not continuous for at least 0.5 seconds. Frequencies were computed using a sliding window of one minute. Activities LiftLowerSquat (5.7, 12.5 kg) and LiftLowerStoop (12.5kg) reached the appropriate peak value, while LiftLowerStoop (5.7kg), LiftExtend (5.7, 12.5kg), and LiftJerky (5.7kg) reached 95%, 64%, 75%, and 88% of their true values, respectively. Data collection was terminated immediately at the end of the test, causing the abrupt stop to the frequency.

Figure 6:

Continuous frequency computed from true activity as labeled manually (solid) and predicted activities filtered (dotted) for SVM classifier and the resulting RNLE Lifting Index.

The frequency of each activity is used to calculate the Frequency Multiplier component of the RNLE (Waters et al., 1994) using the continuous estimations computed by Garg and Kappelusch (Garg and Kappelusch, 2016). The Frequency Multiplier, FIRWL of the detected activity, and load level were combined as shown in Eq. 4 to find Lifting Index over time where LI>1 is risky and LI<1 is safe. All values were calculated for the duration of the test.

$$LI=\frac{Load}{FM\ast FIRWL}$$ (4)

As shown in Figure 6, risk stays below 1 while performing lifts with a safe weight. When unsafe activities or an unsafe weight is used, it increases over 1. During some activities, the FM does not significantly change the FIRWL, giving a constant value. This implies a safe frequency, but an inherently safe/unsafe task.

4. Discussion

The proposed algorithm and system advance the continuous exposure measurements of MMH workers performing MMH tasks. It is an enabling tool that provides detailed information about occupational exposures during prolonged periods of time and can be potentially used to advance our understanding of the exposure dose-injury response relationship for development of occupational LBP.

One of the advantages of the proposed algorithm and system is that it provides information of frequency, duration, and activity type in addition to a single risk value for the activity being performed. This information has an important practical application as it can be used to guide task scheduling for workers with varying time scales, for example, hourly, daily, or weekly. It expands on previous research by computing a real-time single variable of risk with the RNLE Load Index, which takes into account the activity type, load level, and frequency. The duration of individual activities can also be easily determined from frequency detection for later analysis, however, this value is less useful for risk assessment as the duration is pre-defined as the test length or work shift length.

This research primarily builds on previous research by Antwi-Afari et al. (2018, 2019), where instrumented insoles were used to classify a similar set of individual activities. It improves the previous research work by additionally detecting the load level of the activity and computing the frequency of the activity. More recent research by Antwi-Afari et al. (2020) also computes a frequency over time and load level, but does not classify activities individually, similar to the approach of Donisi et al. (2021) who used a single accelerometer to directly classify activities as safe or unsafe. These approaches do not offer any additional information on whether the activity, frequency, or load level caused the risk. This study aimed to improve upon these by first classifying activities and load levels as shown by Lin et al. (2017), then computing and showing a continuous calculation of the potential risk as the frequency changes. This offers real-time advice on when the work needs to slow down, or when the activity or load level is unsafe by itself.

Instrumented insoles have been previously used to detect individual activities when the load level was predefined. However, a changing level of load is important to test for true application to MMH. Recently, varied load levels have been tested by Donisi et al. (2021), but only for the purpose of binary classification (safe/unsafe). Previously, Antwi-Afari et al. (2020) showed that load level values can be directly computed from insole force measurements. This research builds on what has already been experimented with by exploring the possibility of combining load level detection into the machine learning classifier.

This study includes also some limitations, specifically in how the activities were manually labeled. We tried to minimize these potential errors by using the same observer for labeling activities across all participants to minimize the discrepancies. However, due to possible human errors, the exact start and end point of the activities could introduce an error in activity labeling and consequently result in lower classification accuracy.

Another particular limitation of the study was using only two different loads that were relatively small compared to the participant’s mass. Inclusion of various loads, including those greater than 12.5 kg is suggested for future studies as this was out of the scope of this paper. We specifically chose two distinct weights, because a goal of our study was to investigate if using distinctive load classes is a viable approach. The results confirm the validity of this approach, as there was a relatively small difference in the overall accuracy (2%) and the computational cost of distinguishing between carrying weights was small. Participants’ similarity may also influence the conclusions, as all participants were of similar proportions out of the same population. Future studies will remedy this by collecting data from a wider range and larger number of participants, as well as directly from occupational workers.

Finally, risk factors over time were unable to be computed for some activities, so only test sets with continuous performance of activities that had calculable risk were shown for demonstration. Push, pull, and carry all have available counterparts to the RNLE (Liberty Mutual Insurance, 2017; Health and Safety Executive, 2016), however, none provide a succinct risk variable as the Lifting Index of the RNLE does. Future studies will aim to find or create similar values to better form a complete risk assessment method.

The main application of this method is to show when the worker is exceeding recommended criteria. Whether the criteria is exceeded or not, the method additionally provides information about the specific activity being performed, the load level detected, and the frequency of detection. This helps both workers and supervisors pinpoint sources of injury by determining whether an activity is being performed at an unsafe rate or with an unsafe load. Alternately, it can also warn when the activity itself is not feasible regardless of the load and frequency. Additional multipliers can be added to the risk estimation to include different populations (i.e. back injuries, lifting limitations) to further customize this risk quantification to the given set of participants or workers.

In summary, the algorithm can immediately detect hazardous activities (i.e., overextension, jerky lifts, and asymmetric side-to-side lifts) that can be used to provide immediate feedback (i.e., audio, visual, vibrotactile) to the worker. Moreover, the algorithm can provide a cumulative assessment of their exposure through prolonged periods of time and serves as an enabling tool to perform longitudinal studies of ergonomic exposure to workers performing MMH tasks.

5. Conclusion

In this study, we tested a method for collecting data for ergonomic risk estimates using instrumented insoles and a chest mounted accelerometer. Data obtained were 3-axis accelerations, ground reaction forces, and center of pressure coordinates for each foot as well as 3-axis accelerations at the chest. We evaluated a variety of machine learning classifiers to detect a set of MMH activities based on the measurements from these sensors. The best classifier trained was SVM with 85.3% overall accuracy on an independent test set when classifying 18 activities. A negligible difference in classification accuracy was found when distinguishing (18 classes) and not distinguishing (11 classes) activities of different lifting and carrying loads (< 2%). Out of the four different sensors used, combinations including the accelerations and GRF performed better than the rest at activity and load distinction respectively, with the highest accuracy when all data were included. Activity frequencies were continuously computed and tracked over time to calculate the RNLE Load Index, a single variable of risk quantification.

The proposed system enables potential longitudinal monitoring of workers at the workplace through direct measurements. It has the potential to be minimally invasive when using only instrumented insoles, trading a ~4% drop in accuracy for a less obtrusive system - especially when using very thin (~2 mm) and flexible instrumented insoles that are small and flexible enough to minimally affect user comfort and gait. Implementation of our system in real occupational settings will be the aim of our future studies. Successful implementation of our research will help obtain detailed information about the actual exposures of workers that can contribute to advancing our understanding of the exposure dose-injury response relationship in low back pain and help in development of preventative strategies.

Highlights.

• A wearable system for the minimally invasive assessment of biomechanical risk is presented.

• Instrumented insoles provide features to differentiate lifting loads and postures during MMH tasks.

• Hazardous lifting techniques can be classified using machine learning and instrumented insole data.

• Wearable sensors can estimate lifting frequency, duration, and ergonomic risk during MMH.