The SEWAbility system: a video-based job analysis framework for understanding task-specific job demands

Abstract

This study introduces the Smart Evaluation of Work Ability (SEWAbility), an AI-powered video analysis system designed to support objective job analysis by deriving task-specific job demands from real-world sewing task videos. SEWAbility processes overhead surveillance footage from factory environments through a hierarchical analytical pipeline. Videos are first clustered into work tasks using 88-dimensional handcrafted features capturing upper-extremity motion trajectories and joint angle statistics. Each task is subdivided into work cycles and further into work elements, from which Repetitive Motion Pattern (RMP) features are extracted. Based on an analysis of 21 sewing videos across three task categories (A: tops, B: beddings, C: bottoms), SEWAbility effectively distinguished task types. In the video of work task A1, seven work cycles of sewing activity were correctly identified, and 18 repetitive work elements were extracted from the first work cycle. The resulting RMP features provided objective descriptions of motion trajectories, joint speeds, and ranges of motion, yielding interpretable and task-specific biomechanical benchmarks. By enabling scalable and systematic quantification of job demands, SEWAbility demonstrates the feasibility of deriving task-specific biomechanical benchmarks from workplace videos. While this preliminary study analyzed data from a single sewing worker with a physical disability, the broader dataset from more workers will allow future validation. In this way, SEWAbility holds promise to support data-driven employment recommendations and vocational rehabilitation planning for individuals with disabilities.

Supplementary Information

The online version contains supplementary material available at 10.1038/s41598-026-41536-w.

Introduction

Vision-based human motion analysis in work videos

Video analysis has become essential for various domains, including industrial engineering, healthcare, and human activity recognition. In manufacturing, it has supported work method documentation, ergonomic assessment, and time-motion studies, which are traditionally conducted manually using stopwatches or annotations^1,2. Structured video-based behavioral observation techniques have also been applied to assess team dynamics and safety in industrial and organizational settings³. Recent systems like Smart Safety integrate computer vision with big data analytics to enable real-time risk prediction in factory environments⁴. In healthcare, video-based assessments of trauma team performance have demonstrated high observer reliability, even outperforming live evaluations⁵. Advancements in skeleton-based action recognition, including models like ST-GCN, PoseConv3D, and TrajectoryCNN, have shown promising results in capturing complex motion patterns^6–9. In industrial domains such as automobile assembly, transformer-based models and self-supervised learning techniques have enabled automated segmentation of procedures (e.g., oil changes), reducing the need for extensive annotations¹⁰. Similarly, process-mining approaches like “Video-to-Model” and cyclical motion localization frameworks have begun addressing long repetitive tasks in manufacturing to identify work cycles^11,12. Within the garment industry, deep learning–based video analysis has gained increasing attention. Initiatives such as VARSew have demonstrated the feasibility of using convolutional and transformer-based models for real-time action recognition in sewing tasks^13–15.

Despite advancements in video-based motion analysis, its application to job analysis in vocational rehabilitation remains limited due to several practical challenges. Vocational rehabilitation refers to a multidisciplinary, evidence-based process aimed at optimizing work participation for individuals of working age who experience health-related impairments or functional limitations, by systematically addressing work demands, individual capacities, and contextual factors^16,17. However, in the garment industry, concerns such as data privacy, high annotation costs, and inter-worker variability hinder the implementation of current models¹⁴. While recent efforts have focused on action recognition and productivity tracking in sewing tasks¹⁸, existing methods rarely assess individual physical performance to specific job demands.

Job analysis by traditional method

In vocational evaluation, a detailed analysis of workers’ motions during task performance is essential for understanding the physical demands of a job. Traditionally, such evaluations rely on trained professionals who use workplace observations, interviews, or focus groups to assess job requirements¹⁹. As a practical component of this work, we conducted a traditional job analysis at a sewing factory in China during August and September 2024. It involved multiple qualitative methods, including focus group discussions with experienced workers and managers, individual interviews with vocational rehabilitation specialists, on-site workplace observations, and analysis of workplace videos. Each round of job analysis lasted over two hours and included preparing discussion guidelines, facilitating focus group discussions with sewing practitioners, individual interviews with vocational rehabilitation professionals, transcription, and thematic analysis. This traditional job analysis served as a baseline for comparing with the video-based SEWAbility approach.

Building on the conceptual descriptions provided in previous job analysis studies^20–22, this study defines key terms to ensure clarity and consistency throughout the paper: workplace video, work task, work cycle, work element, and repetitive motion pattern (RMP). Detailed definitions and illustrative examples of work task, work cycle, and work element are provided in AppendixA. Workplace video refers to surveillance footage recorded in the production environment, capturing workers performing real tasks at their assigned workstations. In ergonomics, a work task is defined as the basic, meaningful, and functionally complete unit of activity performed by a worker²³. It is regarded as a phenomenon that is scientifically explicable with reference to the work system, including the work process, object, equipment, and materials, as well as the demands placed on the worker, including characteristics, abilities, and skills²⁴. A job typically comprises multiple work tasks²³. For example, in the case of sewing workers, a job may involve a variety of tasks with different types and levels of complexity, such as sewing beddings, tops, or bottoms, depending on the company’s production orders. Within each category of work tasks, workers typically repeat a certain work cycle that consists of a sequence of movements performed in a prescribed order to produce a consistent output²². For instance, stitching bedsheets is a common work task in the sewing industry. A workplace video may capture multiple repetitions of this work task. Each work cycle represents one complete instance of the bedsheet stitching, beginning with actions such as picking up the material and ending with actions such as setting aside the completed work. Within each work cycle are multiple work elements, the smallest meaningful units of motion used in job analysis and in industrial time-and-motion studies²². A closely related concept in the International Organization for Standardization (ISO) terminology is technical actions or actions, defined as elementary manual movements or exertions such as holding, turning, or pushing^25,26. This study adopts the term work elements to ensure consistency with traditional job analysis literature. In the garment industry, work elements may include trunk flexion (e.g., bending down to pick up fabric from the floor), arm elevation (e.g., lifting the fabric off the table), and arm extension (e.g., pushing the fabric forward through the sewing machine needle). In ergonomics, repetitive upper-extremity work involves the repeated execution of similar operations with consistent timing, force patterns, and spatial movements involving the shoulder, arm, or hand²². According to ISO, a repetitive task is characterized by repeating the same work cycles, technical actions, and movements^25,26. While ergonomics studies and standards have acknowledged this concept²⁷, existing definitions typically remain at the task or action level and lack operational criteria suitable for detailed motion analysis. To support more specific descriptions and analysis in this study, we introduce the concept of repetitive motion pattern (RMP) features. Building upon the existing notion of spatial movements, RMP features offer a more granular, data-driven representation of the repetitive work elements within each work cycle. By quantifying biomechanical characteristics, such as joint trajectories, movement speed, and range of motion, RMP features enable structured motion-level analysis of repetitive patterns, supporting precise modeling of physical job demands and enhancing the objectivity of vocational evaluations.

This practical experience highlights that traditional approaches to job analysis are increasingly insufficient to meet the complex demands of modern, customized manufacturing, particularly in the garment industry^28,29. While frontline workers could describe general work tasks such as sewing products according to factory orders, they struggled to articulate the detailed structure of a work cycle. Factory managers tended to focus on production targets rather than the physical capacities required for task completion. Workers and managers often remarked that they “know how to do it but cannot describe it in detail,” whereas vocational rehabilitation professionals could identify key work elements, such as repetitive arm movements and bending to pick up fabric, but were unable to quantify them in terms of speed or range. These findings reveal the inherent limitations of traditional job analysis: it often lacks specificity, is prone to subjective bias, and varies with assessor expertise, cultural interpretation, or the worker’s ability to verbalize their tasks^24,30,31. Against this backdrop, the vagueness of traditional job analysis outcomes underscores the need for more objective and data-driven approaches, including analyzing work cycle patterns of the work task, exploring work elements, and calculating RMP features from workplace videos.

To our knowledge, no current system offers a video-based solution for capturing and analyzing workers’ motion data to derive objective, task-level job demands tailored for use in vocational rehabilitation. This study aims to introduce a novel approach that integrates artificial intelligence (AI) and video-based analysis into the traditionally manual field of job analysis. We propose a vision-based framework, SEWAbility (Smart Evaluation of Work Ability), aiming to extract interpretable and quantifiable job demands from real-world workplace videos and to obtain task-specific RMP features that serve as reference benchmarks for evaluating individual work capability in vocational rehabilitation settings.

Proposed methodology

This section describes the methodology adopted in the SEWAbility system. Section 2.1 provides an overview of the system pipeline. Section 2.2 introduces the dataset. Section 2.3 details the data processing steps.

Pipeline overview of SEWAbility system

Figure 1 illustrates the stepwise pipeline for decomposing and analyzing sewing-related motion, progressing from raw workplace videos to fine-grained job demand extraction. Each stage builds upon the previous one, guided by defined clustering strategies, segmentation rules, or feature computation methods.

Fig. 1.

Pipeline Overview of Smart Evaluation of Work Ability (SEWAbility) System.

Dataset

We selected the sewing industry as the application domain due to its global economic importance and high employment capacity for marginalized populations^32–34. Sewing tasks are performed in stable, repetitive settings at fixed workstations, making them well-suited for unobtrusive video-based analysis³⁵. The research took place at Hunan TuoFu Textile Co., Ltd. in Yueyang, China, coordinated by the Hunan Rehabilitation Association of People with Disabilities. In this factory, sewing workers use the DOLLOR Single Stepper High-Speed Computerized Lockstitch Sewing Machine (Model: B3A). It activates with minimal pedal pressure, reducing ankle movements and lowering lower limb strain. This ergonomic design improves comfort and accessibility, shifting physical workload to the upper extremities, which our study analyzes.

A total of twenty-one sewing workers consented to the recording and use of their work videos for data collection in this project. Among them, seven had physical disabilities, two had intellectual disabilities, five had hearing and speech impairments, and seven had no disabilities. The participants’ ages ranged from 21 to 61, with an average age of 45.9. Their work experience varied from 2 to 43 years, with a mean of 19.7. As all had over one year of continuous employment, they were classified as being in stable employment under Chinese policy³⁶. Written informed consent was obtained from all participants prior to data collection. All study procedures were conducted in accordance with the relevant guidelines and regulations and complied with the Declaration of Helsinki. The data collection procedures, including camera installation and video recording, were reviewed and approved by the Department of Rehabilitation Sciences Ethics Committee at The Hong Kong Polytechnic University (Ethics Approval Code: HSEARS20240815005).

Workplace videos were recorded with a high-resolution network camera that features and captures 3840 × 2160 pixels. Positioned 1.2 m above the workstation, it provides a clear top-down view of upper extremity movements while avoiding task interference and maintaining privacy, as faces are invisible. Although the overhead cameras have been continuously recording since September 24, 2024, only selected segments were used for this preliminary analysis. As of September 2025, recording is still ongoing. A review of collected workplace videos revealed that the factory schedules different sewing tasks based on production demands, including school uniforms, household bedding, medical gowns, hospital garments, etc. Frame rates are adaptive, generally fluctuating between 15 and 25 FPS based on network bandwidth and scene complexity. For this exploratory study on integrating AI into vocational evaluation, we focused on a single participant, a 25-year-old male sewing worker with a physical disability and three years of experience. The dataset used consisted of 21 videos representing three common categories of real-world sewing tasks. sewing tops (A1–A6), sewing beddings (B1–B7), and sewing bottoms (C1–C8). These categories were selected based on previous qualitative findings from this factory, reflecting the most frequently performed tasks in actual production settings. Sewing tops (A) typically requires frequent handling of smaller fabric pieces, more posture changes, and finer upper-limb adjustments. Sewing beddings (B), by contrast, mainly involves long straight seams on relatively large, heavy fabric panels, leading to wider arm reaches and repetitive forward trunk flexion. Sewing bottoms (C) is the most complex of the three tasks, involving multiple fabric layers, curved seams, and various stitching angles, which place higher demands on coordination and sustained hand-wrist control. These task-specific characteristics informed the choice of spatiotemporal and kinematic features used in the analysis. A summary of the dataset is presented in Table 1.

Table 1.

Dataset Description.

Video ID	Work Tasks	Duration (min: sec)	FPS	Resolution
A1.mp4	Tops	7:11	25.0	3840 × 2160
A2.mp4	Tops	7:12	25.0	3840 × 2160
A3.mp4	Tops	7:12	25.0	3840 × 2160
A4.mp4	Tops	7:12	25.0	3840 × 2160
A5.mp4	Tops	7:12	25.0	3840 × 2160
A6.mp4	Tops	8:41	22.3	3840 × 2160
B1.mp4	Beddings	9:03	15.0	3840 × 2160
B2.mp4	Beddings	9:00	15.0	3840 × 2160
B3.mp4	Beddings	9:00	15.0	3840 × 2160
B4.mp4	Beddings	9:00	15.0	3840 × 2160
B5.mp4	Beddings	9:00	15.0	3840 × 2160
B6.mp4	Beddings	9:00	15.0	3840 × 2160
B7.mp4	Beddings	9:00	15.0	3840 × 2160
C1.mp4	Bottoms	5:03	15.0	3840 × 2160
C2.mp4	Bottoms	9:00	15.0	3840 × 2160
C3.mp4	Bottoms	9:00	15.0	3840 × 2160
C4.mp4	Bottoms	9:00	15.0	3840 × 2160
C5.mp4	Bottoms	8:59	15.0	3840 × 2160
C6.mp4	Bottoms	9:00	15.0	3840 × 2160
C7.mp4	Bottoms	9:00	15.0	3840 × 2160
C8.mp4	Bottoms	9:00	15.0	3840 × 2160

Processes of SEWAbility

The recorded videos were processed through a multi-step workflow, including Work Task Clustering, Work Cycle Segmentation, Work Element Extraction, and RMP Feature Computation.

Work task clustering

The SEWAbility system employs the MediaPipe Pose solution to extract upper-extremity joint coordinates from workplace videos. For each video fragment T_i, we extracted an 88-dimensional feature vector (F_i), which represents the motion features based on the trajectory statistics of the wrist, elbow, and shoulder, as shown in Eq. (1).

1

The 88 features are composed of two groups: 72 track and 16 range-of-motion features. The first 72 dimensions capture motion trajectory statistics of six upper-extremity joints (bilateral shoulders, elbows, and wrists) in X and Y directions, using six descriptors: average speed, speed variability, amplitude, dominant frequency, energy, and spectral centroid. The remaining 16 dimensions encode joint range-of-motion features for four upper-extremity joints (bilateral shoulders and elbows), including angle mean, angle standard deviation, angle velocity change mean, and angle velocity change standard deviation.

Thus, the complete representation is as follows:

2

All videos were then aggregated into a set, followed by K-means clustering shown as follows:

3

where each cluster C_k groups videos with similar overall motion patterns. We referred to the silhouette coefficient to evaluate clustering quality. Further, we incorporated domain-specific knowledge of task structures to determine the final number of clusters K. This process completes clustering workplace video fragments into distinct work task categories.

Work cycleSegmentation

Prior studies suggest that work cycle segmentation can be effectively guided by spatial or behavioral cues, such as transitioning between workstation zones or initiating a repetitive action, rather than relying solely on time-based methods³⁷. For example, in manual assembly work, spatial transitions within predefined workstation zones have been used to define cycle boundaries, where a new cycle starts when the operator leaves the assembly area to retrieve new materials¹. This spatial cue-based segmentation approach has demonstrated practical value in structured industrial settings.

In the context of sewing tasks, the collected videos revealed that bending down to change fabric is a consistent and observable behavior that typically marks the beginning of a new work cycle. To capture this behavior programmatically, we adopted a vision-based approach that leverages the spatial constraints inherent in the workstation layout. Specifically, we observed that sewing workers consistently perform a “fabric-changing” action at the end of each work cycle: they bend down to place completed fabric beside the sewing table and then retrieve new material from the ground to initiate the next task. During this transition, only the right arm typically remains visible, resting on the table edge, while the left arm temporarily exits the camera’s field of view. Based on this consistent behavioral pattern, we defined a polygonal region of interest (ROI) covering the edge of the sewing table. We continuously tracked the right wrist joint frame by frame using pose estimation. A work cycle boundary was identified when the right hand was detected within the ROI, and no hand appeared outside the ROI, for a continuous period of at least one second. This condition marks the completion of a work cycle, reflecting the typical behavior of placing the finished fabric aside before starting the next work cycle.

This rule-based ROI detection approach is advantageous for industrial settings where workstations are spatially constrained and task transitions exhibit consistent physical cues. It provides a lightweight, annotation-free method for reliable work cycle segmentation in real-world factory video data. Formally, given a task video T_i, we denote L_c+1 detected boundary timestamps where each L_i satisfies the above ROI-based condition and temporal stability.

Each work cycle is then defined as the interval corresponding to a complete unit of repeated task execution:

4

Therefore, T_i is segmented into c work cycles, denoted as follows:

5

Work elementextraction

To extract work elements within each work cycle, we employ a two-step segmentation method: a time-based approach (i.e., sliding window segmentation) and a spatial approach based on wrist joint trajectories (i.e., dynamic motion-peak window segmentation).

Previous research demonstrated that a 1-second window is large enough to capture meaningful motion dynamics while maintaining smoothness in temporal patterns¹. Prior studies in human activity recognition have also demonstrated that a window size of 20–25 frames provides optimal accuracy and processing speed compared to longer window lengths³⁸. To balance temporal resolution and computational efficiency, we apply a sliding window approach that segments the work cycle W_j into fixed-length overlapping segments (1-second window, 0.5-second step). The equation is as follows:

6

Then, each segment S_{j, I} is represented by an 88-dimensional feature vector as follows:

7

These segments are then grouped using K-means clustering as follows:

8

We identify the dominant motion pattern within this cycle by selecting the most frequently occurring cluster. The equation is as follows:

9

To ensure temporal consistency, we extract the consecutive sequence of segments belonging to cluster C_j^∗. Let the corresponding index set by the following equation:

10

The representative fragment V_j is then defined by concatenating these consecutive segments, as shown in following equation:

11

This segment, Vj, is used for subsequent work element extraction. We employ a dynamic segmentation approach where segment boundaries are determined by detecting local motion peaks in the wrist joint trajectory. The wrist joint was chosen because of its greater movement range and key role in sewing task execution compared to more stable proximal joints³⁹. This adaptive method allows segment durations to naturally align with the rhythm of repetitive motions, improving boundary localization accuracy. Previous research also demonstrated that dynamic windowing enhances work element segmentation and recognition performance⁴⁰. V_j is the concatenated sequence of consecutive segments from cluster C_j^∗ from the previous step. V_j is further segmented using motion peak-based segmentation by the wrist joint trajectory to analyze repetitive work elements.

Using the motion trajectory of wrist joints after a Gaussian smoothing filter to eliminate noise, we detect local motion peaks that indicate repetitive motion boundaries. This yields a variable-length segmentation, shown as follows:

12

Each segment M_j corresponds to a single work element from the videos composed with the work cycle’s most frequently occurring sliding window cluster. After segmentation, we evaluate each work element’s degree of repetition by computing two indicators: (1) a periodicity score derived from autocorrelation to capture the strength of repeating motion patterns, and (2) a spectral concentration score based on the ratio of dominant-frequency energy to the remaining spectrum (signal-to-noise ratio). We examined the temporal sequence of work elements and retained those with scores similar to at least one adjacent work element, ensuring that they represented locally consistent repeated motions rather than isolated outliers. Among these, the work elements with the highest frequency of occurrence within the work cycle were identified as repetitive work elements N_j. The features of the repetitive work elements N_j were extracted and analyzed to construct job demand profiles.

RMP feature computation

For each repetitive work element N_j, we computed a set of features to describe it, named RMP features. These features are derived from joint positions (shoulders, elbows, wrists), velocities, angles, and motion intervals. Unlike previous fixed 88-dimensional feature vectors, these descriptors are tailored to characterize trajectories and range-of-motion features from vocational rehabilitation perspectives.

Let h_{j, l} denote the feature vector for the motion l segment in V_j, where d denotes the feature dimension, including aspects of joint trajectory and range of motion. The equation is as follows:

13

The extracted features from all segments within V_j are aggregated as follows:

14

Finally, to construct the complete dataset, we collect all feature vectors across work cycles to establish the RMP feature profile, as shown in the following equation:

15

Experiment results

This study analyzed a subset of video data captured from actual sewing workstations to demonstrate the feasibility and potential of the SEWAbility system. The experiment result section describes the data analysis and results, including Results of Work Task Clustering (3.1), Results of Work Cycle Segmentation (3.2), Results of Work Element Extraction (3.3), and Results of RMP Feature Calculation (3.4).

Results of work task clustering

We conducted an initial analysis to examine whether global video-level motion characteristics could effectively distinguish between different task types. Following the SEWAbility pipeline, an 88-dimensional feature vector capturing spatiotemporal trajectories and joint angle descriptors was extracted for each video. We then applied K-Means clustering to group the workplace videos based on their global motion characteristics, with the optimal number of clusters determined to be K = 4.

As illustrated in Fig. 2, the 21 videos were partitioned into four clusters. Given the high dimensionality of the feature space (88 features), we visualized the clustering results in a two-dimensional space using the first two principal components from principal components analysis (PCA). Of the 21 videos, 18 were correctly clustered into their corresponding task categories, with misclassifications occurring for B2, C4, and C5. Work task clustering achieved a Hungarian-aligned accuracy of 85.7% (18/21). Misassignments were primarily due to a small split of the C class (C4 and C5 isolated as a separate cluster), and a single B video (B2) mixed into the C-dominant cluster. The results support the suitability of the 88-dimensional features for distinguishing sewing task types. The clustering closely matched the task categories, indicating that the extracted motion descriptors effectively capture task-specific differences. These features are therefore appropriate for further modeling and analysis.

Fig. 2.

Work task clustering PCA Plot.

Results of work cycle segmentation

We then applied the whole data processing pipeline to a representative video, A1, a task of sewing tops. Following the previously described method, eight transition points were identified based on hand positions relative to the predefined ROI, as shown in Fig. 3.

Fig. 3.

Transition points for work cycle segmentation.

(Images in this figure are screenshots extracted from workplace videos recorded by the first author, Huiling Hu.)

These nine transition points in A1 were used to segment the video into eight complete work cycles. As shown in Table 2, each cycle’s start and end frames correspond to the detected temporal boundaries. All work cycles in A1 were correctly identified and compared to those in the manual review, in which three trained volunteers recorded work cycles in each video and the number of repetitive work elements within each cycle across 21 videos. The RMSE (Root Mean Squared Error) for this work cycle segmentation count was 1.29 for work task A, 4.96 for work task B, and 8.65 for work task C, with an overall RMSE of 6.10 across all groups. A further boundary-tolerant evaluation (± 2 s window) was conducted on six videos of work task A (A1–A6). Precision, recall, and F1-scores were 1.00 for A1 and A2, indicating perfect alignment between detected and manually annotated work cycle boundaries. However, performance declined in A3–A6, with F1-scores of 0.25 (A3), 0.31 (A4), 0.18 (A5), and 0.57 (A6). The drop in accuracy for A3–A6 was primarily due to misclassifying specific within-task actions as “fabric-changing” events, which serve as work cycle boundaries in our method. These false positives caused extra, incorrect segmentation points, while some true boundaries were missed, reducing precision and recall. This indicates that the segmentation algorithm performs very well under stable and distinct boundary actions (A1–A2) but is more prone to errors in videos where task actions occasionally resemble the fabric-changing motion, leading to boundary confusion.

Table 2.

Work Cycle Segmentation.

Cycle	Start_Frame	End_Frame	Start_Time	End_Time	Duration (sec)
1	532	1805	0:21	1:12	50
2	1805	3118	1:12	2:04	52
3	3118	4216	2:04	2:48	43
4	4216	5424	2:48	3:36	48
5	5424	6780	3:36	4:31	54
6	6780	7994	4:31	5:19	48
7	7994	9177	5:19	6:07	47
8	9177	10,408	6:07	6:56	49

Results of work element extraction

To conduct work element extraction, we first applied a 1-second sliding window with a 0.5-second step for each identified work cycle to generate overlapping motion segments. Each sliding window segment was encoded as an 88-dimensional feature vector and clustered using K-means (K = 3).

Take A1 as an example, this process revealed Cluster 0 dominating the first seven (Fig. 4a), for example, 84.8% in work cycle 1 (Fig. 4b). Work cycle 8, showing a markedly different distribution, was deemed an outlier (Fig. 4c); manual inspection confirmed no sewing activity, and it was excluded from further analysis.

Fig. 4.

Cluster distribution in Work cycles.

Based on this observation, we extracted the continuous sequence of Cluster 0 segments to identify RMPs. We then applied smoothing to the bilateral wrist trajectories across these consecutive Cluster 0 segments. Take work cycle 1 as example, Fig. 5 shows the smoothed average trajectories of the left and right wrists in both the X and Y directions, with vertical dashed lines indicating the work element segmentation points.

Fig. 5.

Work element segmentation.

As a result, within Work Cycle 1, we detected 30 work elements. Applying the repetition-regularity filtering, we retained the 18 repetitive work elements, with an average duration of 1.08 s. The same procedure was applied to Work Cycles 2–7. Compared with manual annotations, the automated method yielded an RMSE of 2.19 work elements per work cycle, indicating that average predicted work element counts deviated from manual review by about two work elements. Since a typical work cycle includes about a few dozen work elements, this deviation is relatively minor.

Results of RMP feature computation

The 18 repetitive work elements identified within Work Cycle 1 were analyzed, and a complete set of motion features was extracted to establish the RMP feature profile, capturing the typical range and variability of task performance. To illustrate the diversity within this work cycle, six representative work elements were selected based on their amplitude ranking: the two work elements with the lowest amplitude (16th and 17th), two with typical amplitude (4th and 10th), and two with the highest amplitude (2nd and 5th). The bilateral wrist trajectories for these six segments are shown in Fig. 6.

Fig. 6.

The wrist trajectory of the representative work elements.

Thirty-six quantitative descriptors of bilateral upper extremities were extracted from each segment to characterize the RMP features. These included 24 trajectory-based features representing motion range and average speed along the x and y axes for bilateral shoulder, elbow, and wrist joints, and 12 range-of-motion features capturing the minimum, maximum, and mean of joint angles for bilateral shoulders and elbows. These detailed motion profiles provide a quantitative basis for understanding task-specific job demands.

The worker’s shoulder width, defined as the horizontal distance between the left and right shoulder joints, was set to 50 cm and used as a reference scale to enable spatial normalization and real-world interpretability. The eighteen right-side upper extremity descriptors were used as illustrative examples. One of the highest-amplitude work elements (2nd) was selected for detailed visualization and quantitative analysis, as shown in Table 3.

Table 3.

The repetitive motion pattern features.

Item	Data
right_shoulder_x_range_cm	4.65
right_shoulder_x_speed_cmps	7.17
right_shoulder_y_range_cm	4.25
right_shoulder_y_speed_cmps	7.04
right_elbow_x_range_cm	8.32
right_elbow_x_speed_cmps	11.97
right_elbow_y_range_cm	4.37
right_elbow_y_speed_cmps	10.55
right_wrist_x_range_cm	11.6
right_wrist_x_speed_cmps	12.88
right_wrist_y_range_cm	12.51
right_wrist_y_speed_cmps	22.57
right_elbow_angle_max	126.67
right_elbow_angle_mean	99.65
right_elbow_angle_min	76.37
right_shoulder_angle_max	137.64
right_shoulder_angle_mean	128.12
right_shoulder_angle_min	119.93
Segment_Index	2
Start_Frame	886
End_Frame	926
Start_Time	0:35.44
End_Time	0:37.04

Discussion

In the present paper, we proposed the development of SEWAbility, an AI-enhanced video analysis approach for assessing task-specific job demands in sewing work. The results demonstrated that SEWAbility can objectively identify and quantify the repetitive and biomechanical characteristics of sewing tasks. Using real-world sewing workplace videos, SEWAbility was able to cluster work tasks, segment work cycles, extract work elements, and compute RMP features. SEWAbility represents a significant advancement in vocational rehabilitation by introducing a data-driven, objective, and scalable method of analyzing task-specific job demands. This method facilitates the development of criteria-based reference benchmarks for evaluating individual work capability, thereby supporting more accurate and inclusive job matching. The accuracy of work task clustering (85.7%) suggests that global motion features captured by the SEWAbility system have the potential to distinguish between different types of sewing tasks, even in an unsupervised setting. Using sliding window segment cluster analysis, the SEWAbility system first detected dominant motion patterns. As shown in Fig. 4, Cluster 0 consistently dominated across work cycles, accounting for 84.3% of segments in Work Cycle 1. This high proportion of repeated motion patterns strongly supports the classification of the task as repetitive manual work. According to the draft revision ISO/DIS 11228-3:2025, a task is considered repetitive if similar actions are repeated for more than 30% of its duration²⁶, while earlier ergonomic guidelines use a 50% threshold²². Figure 5 further illustrates the periodicity of motion using joint trajectory waveforms. Notably, the X-direction trajectories (orange line) showed greater fluctuation than the Y-direction trajectories (blue line), which remained relatively stable. This pattern corresponded with both Fig. 6 and manual video review, where workers repeatedly retracted and extended their arms to push fabric horizontally through the sewing machine, corresponding to the actual sewing motion in which workers frequently performed arm extension and retraction in the X-direction and rarely performed adduction or abduction movements in the Y-direction. The pronounced horizontal movements explain the stronger periodicity observed along the X-axis. The detailed RMP features in Table3 provide a rich and precise representation of job demand profiles. These data were calculated using the participant’s shoulder width of 50 cm in this video. In future experiments, a reference object of known length could be placed on the workbench to calibrate spatial measurements further.

The system responds to a critical gap in traditional job analysis by profiling job demands in a structured, quantifiable, and interpretable manner, which is essential for evidence-based vocational rehabilitation practice. By continuously recording sewing workers’ movements and extracting task-specific RMP features, the system generates standardized job demand profiles that serve as reference benchmarks for evaluating applicants’ physical work ability. Within the context of vocational rehabilitation, such objective job demand modeling supports more accurate matching between individual functional capacity and task requirements, particularly for individuals with health-related impairments or limited work experience. This evidence-based approach shows promise in improving evaluation efficiency by reducing reliance on labor-intensive expert observation and enabling scalable assessments across workers and tasks. It also enhances objectivity by minimizing assessor bias and ensuring standardized, reproducible, and transparent evaluations, which are critical challenges in vocational rehabilitation decision-making. In doing so, SEWAbility introduces a novel paradigm for data-driven job matching that aligns with Human-Centered AI principles of explainability, expert-in-the-loop decision-making, and adaptability⁴¹. For employers, this enables more informed recruitment and task allocation; for job seekers, especially persons with disabilities or those with limited work experience, it offers a fairer and performance-based pathway to employment; for vocational rehabilitation professionals, it provides objective decision support for job placement, accommodation, and return-to-work planning. Beyond individual applications, the detailed motion data generated by SEWAbility could also strengthen national occupational standards, such as China’s National Occupational Classification, which currently lacks task-specific biomechanical descriptors³⁴.

This study extends AI applications in vocational rehabilitation by moving beyond outcome prediction to direct task-level evaluation. Previous AI-based approaches in vocational rehabilitation have concentrated mainly on predicting return-to-work outcomes, estimating recovery durations, or identifying risk factors for prolonged disability^42–44. For instance, Smart Work Injury Management systems have demonstrated the potential of AI by improving predictions of sick leave durations and recovery trajectories through neural network models^45–47. In contrast, SEWAbility leverages video-based motion analysis to capture the physical and temporal requirements of work tasks, thereby extending AI applications beyond outcome prediction to task-level evaluation. This innovation bridges methods from industrial engineering with vocational evaluation needs, offering a novel pathway to assess and model job demands objectively.

The sewing industry provides a relevant application domain where SEWAbility shows potential to support both workforce sustainability and inclusive employment. As a labor-intensive sector with high global relevance, sewing tasks are well suited for unobtrusive video-based analysis, offering scalable solutions that support both workforce demand and opportunities for workers with disabilities. The textile sector, including sewing operations, is a cornerstone within global supply chains, contributing substantially to GDP and employment worldwide. The global textile market, valued at approximately USD 1.98 trillion in 2024, is projected to experience sustained growth driven by globalization and offshoring, significantly benefiting employment among women and marginalized groups, such as those with limited educational backgrounds or physical impairments^33,34,48. Despite China’s continued dominance in global textile production and export, its domestic labor market faces persistent shortages in the sewing industry, while people with disabilities remain underrepresented in the workforce^32,49,50. SEWAbility provides a promising framework that may help address the dual challenges of labor shortages and inclusive employment in the textile industry. The system unobtrusively monitors workstations with the sewing worker. Its non-intrusive nature allows for prolonged, unbiased data collection without interfering with the sewing worker, such as wearable sensors or manual on-site observation and assessments¹.

Limitations and future work

Although the SEWAbility system shows significant promise, several limitations should be recognized. First, the current dataset comes from a single sewing factory, which may limit how well the findings apply to other industrial settings or different job roles. Additionally, the study only analyzed videos of skilled and consistently employed workers who performed tasks successfully. This sampling method might not fully represent the range of worker performance, especially since it lacks negative examples from individuals unable to meet task requirements. As a result, the model has limited exposure to poor or error-prone performances, reducing its ability to predict job seeker qualifications accurately. Second, some videos also showed moments where workers temporarily left their workstations or engaged in unrelated activities, such as using a mobile phone. Occasional video stuttering and frame drops were observed, introducing noise and suggesting that initial data cleaning might be needed. Third, the model’s generalization ability across different sewing tasks remained uncertain. For instance, the RMSE during work cycle segmentation varied widely, showing significantly poorer performance in the work task B and C. This variability likely stemmed from the greater complexity of these tasks, indicating a need to improve the segmentation method. Another factor that affected segmentation accuracy was the use of the MediaPipe Pose model, selected for its speed and lightweight design, which occasionally misclassified within-task movements as fabric-changing actions and led to over-segmentation.

Building on SEWAbility’s demonstrated potential and addressing its current limitations, the dataset should be expanded to capture how different workers, both with and without disabilities, perform the same task, enabling a more comprehensive representation of task execution variability. Implementing initial data-cleaning procedures and adopting more accurate pose estimation models, such as OpenPose or ViTPose, will also improve the accuracy of key system components, especially in detecting “fabric-changing” actions to segment work cycles, thereby enhancing the overall reliability and generalizability of the system model. Once sufficient annotated data are available, we plan to conduct a systematic evaluation of the algorithm’s predictive accuracy using more widely adopted machine learning metrics, such as mean Intersection over Union (mIoU), or Mean Absolute Error (MAE). These metrics will enable us to assess the system’s ability to cluster work tasks, identify work cycles, and correctly compute the repetitive work elements. The dataset will be split into training and validation sets to ensure robust performance estimates, with cross-validation strategies applied during model development.

Beyond technical metrics, it is also essential to assess the practical value of SEWAbility in real-world job analysis and vocational evaluation contexts. A key area for the future is developing a unified framework that combines objective job analysis with work capacity evaluation (WCE). Current WCE systems lack effective ways to simulate different work tasks and are limited in precision, especially for remote or underserved populations^51,52. We are developing a performance-based work assessment station using Virtual Reality (VR), Mixed Reality (MR), or Extended Reality (XR) technology to address this gap. This innovative approach enables users to perform standardized, simulated work tasks within an immersive environment. During evaluations, their motion data was recorded and quantitatively compared against RMP features and job demand profiles established from real work videos. SEWAbility could function as a decision support system to determine users’ capacity to meet the physical job demands of sewing tasks. Discrepancies can be used to guide targeted interventions, and longitudinal follow-up can track whether these interventions translate into improved work capacity. By integrating both quantitative performance evaluation and real-world applicability testing, we aim to establish SEWAbility as a technically sound system and a clinically relevant and practically useful tool for enhancing person–job matching in vocational rehabilitation.

Conclusion

This study introduces SEWAbility, a video-based computational framework designed to analyze task-specific job demands using workplace videos from the sewing industry as a demonstration. The system identifies task-specific RMP features by leveraging human pose estimation and handcrafted motion features and extracts detailed kinematic descriptors across the complete work cycles. The results demonstrate SEWAbility’s capability to differentiate between sewing task types and to generate quantitative benchmarks at the work element level. The RMP features derived from this system provide a structured, objective, and interpretable basis for understanding physical job demands. Compared to traditional job analysis approaches, which often rely on subjective observation and interviews, SEWAbility offers consistent and fine-grained motion analysis over extended periods. The findings from this study have the potential to inform the scientific rigor of job matching while also enabling the future design of WCE and targeted functional rehabilitation programs.

Supplementary Information

Below is the link to the electronic supplementary material.

Author contributions

Conception and design of the work: Andy Cheng, Peter Ng, Huiling Hu. Resources, data curation, formal analysis and investigation, writing-original draft preparation: Huiling Hu. First round review of proposed methodology and implementation status: Zhaoyun Ding, Eugene Yujun Fu. Second round review of full text, final approval of the version to be submitted and published, project supervision: Peter Ng, Andy Cheng.

Funding

This study was supported by Dr. Peter Ng’s research project funded under the Research Matching Grant Scheme (Project No. P00052481), administered by the University Grants Committee of the Hong Kong Special Administrative Region Government.

Data availability

The raw video data used in this study cannot be publicly shared due to privacy and ethical considerations, as the data contain identifiable individuals who did not consent to public release.The complete analysis code underlying this study is publicly available. The source code is maintained in a public GitHub repository (https://github.com/Ice-HL/SEWAbility.git). This ensures long-term accessibility and reproducibility of the proposed method.

Declarations

Competing interests

The authors declare no competing interests.

Ethics approval and consent to participate

The study was reviewed and approved by the Department of Rehabilitation Sciences Ethics Committee at The Hong Kong Polytechnic University (Ethics Approval Code: HSEARS20240815005). All study procedures were conducted in accordance with the relevant guidelines and regulations and complied with the Declaration of Helsinki. Prior to participation, all students were informed about the web camera recording and assured of anonymity. Data and personal identities were kept strictly confidential. Written informed consent was obtained from all participants.

Consent for publication

All participants consented to publish the data and any accompanying images and videos. All authors have read and approved the manuscript for publication.

Confidentiality

The recorded dataset was kept strictly confidential and stored securely throughout the study. Access was restricted solely to the authors by ethical guidelines.

Appendix

Appendix A Definitions and Examples of Key Terms