Hand pose selection in a bimanual fine-manipulation task

Abstract

Many daily tasks involve the collaboration of both hands. Humans dexterously adjust hand poses and modulate the forces exerted by fingers in response to task demands. Hand pose selection has been intensively studied in unimanual tasks, but little work has investigated bimanual tasks. This work examines hand poses selection in a bimanual high-precision-screwing task taken from watchmaking. Twenty right-handed subjects dismounted a screw on the watch face with a screwdriver in two conditions. Results showed that although subjects used similar hand poses across steps within the same experimental conditions, the hand poses differed significantly in the two conditions. In the free-base condition, subjects needed to stabilize the watch face on the table. The role distribution across hands was strongly influenced by hand dominance: the dominant hand manipulated the tool, whereas the nondominant hand controlled the additional degrees of freedom that might impair performance. In contrast, in the fixed-base condition, the watch face was stationary. Subjects used both hands even though single hand would have been sufficient. Importantly, hand poses decoupled the control of task-demanded force and torque across hands through virtual fingers that grouped multiple fingers into functional units. This preference for bimanual over unimanual control strategy could be an effort to reduce variability caused by mechanical couplings and to alleviate intrinsic sensorimotor processing burdens. To afford analysis of this variety of observations, a novel graphical matrix-based representation of the distribution of hand pose combinations was developed. Atypical hand poses that are not documented in extant hand taxonomies are also included.

NEW & NOTEWORTHY We study hand poses selection in bimanual fine motor skills. To understand how roles and control variables are distributed across the hands and fingers, we compared two conditions when unscrewing a screw from a watch face. When the watch face needed positioning, role distribution was strongly influenced by hand dominance; when the watch face was stationary, a variety of hand pose combinations emerged. Control of independent task demands is distributed either across hands or across distinct groups of fingers.

INTRODUCTION

Humans are capable of performing a variety of tasks where their two hands cooperate and complement each other; everyday examples include cutting a steak with knife and fork or opening a bottle cap. Many crafts require exquisitely fine coordination of both hands, from stitching to surgery and watchmaking, where the manipulated tools can be extremely small and, thus, manipulation often requires a microscope. Acquisition and fine-tuning of such skills necessitates many years of practice. In fact, improvements probably never stop, as suggested by a seminal cross-sectional study by Crossman (1) reporting data over many years of experience. When performing bimanual skills, both hands have to adopt intricate poses to cooperatively maneuver target objects with balanced forces and torques applied to achieve the task goals. How the human central nervous system (CNS) controls all the degrees of freedom of the two hands to adopt appropriate hand poses and finger positioning is still poorly understood.

Hand Pose Selection for Tool Use

A wealth of evidence shows that the fingers’ postures are determined before the hand closes on objects through the preshaping of fingers, reflecting synergistic control of the fingers (2). Hand pose selection for tool use is task dependent, and the same tool may also be held differently depending on the specific subtask. The selection of hand poses is guided by a variety of factors, including the objective of the manipulation, the task’s requirements and the physical properties of the manipulated object (3), all of which we deem as task demands. Proper choice of hand pose is crucial to task performance. Multiple lines of evidence indicate that our brain recognizes task demands before movement execution (4) and that hand pose selection is influenced by the objective of the manipulation (5) and the forces required by the task (6). For instance, power grasps that require both fingers and palms are indicative of tasks requiring high stability or large forces such as wringing a towel or screwing the cap of a jar. Conversely, pinch grasp poses that use primarily fingertips are typical of tasks requiring fine motor skills, such as writing with a pencil or playing musical instruments.

Human hand pose selection before grasping an object is also guided by the purpose of the subsequent manipulation, its end-goal (7), the anticipated end-state (8), and the dynamics of grasping (9). It is also associated with the target’s physical properties, such as size (10), shape (11, 12), orientation (13), and spatial location (14). Priming of the grasping kinematics has been studied in a variety of task scenarios, including the grasp-to-use and the grasp-to-move (15). Longer reaction times were reported in the former scenario, indicating a more extensive movement planning process (16). The ability of selecting task-oriented hand poses can be improved with practice (17).

From Unimanual to Bimanual Hand Pose Selection

Although there has been a vast literature on the role of human hand poses for tool use, the vast majority of studies focused on unimanual shaping of the fingers, in particular, grasping. When the task requires both hands to manipulate the same object, shaping of each hand pose cannot be considered in isolation. Both hands will be recruited in such a way to balance roles and efforts in a symmetric or asymmetric manner. Symmetric and balanced roles are observed when the two hands perform identical motion, e.g., lifting a heavy box. Balanced role distribution has received more attention and has also been studied extensively in rhythmic movements (18–20).

The asymmetric role distribution across the two hands and how these roles change in response to different task demands remains an open challenging question. For example, a knife is held differently when one cuts an apple versus when one peels an apple. Furthermore, the dominant and nondominant hand adopt different roles. To cut an apple in half, one typically uses the dominant hand to operate the knife, whereas the nondominant hand assists by holding the apple in place. Similarly, when peeling an apple, the fingers of the assisting hand rotate the apple to allow the dominant hand to cut the apple. Tasks that require asymmetric and complementary roles between the hands have been referred to as role-differentiated bimanual manipulation tasks (21, 22). The dominant hand controls the part of the task that requires higher dexterity and efficiency, whereas the assisting hand plays an auxiliary role by stabilizing the object (23–25).

Handedness originates from hemispheric differences of the human brain (26). Each hand is controlled predominantly by the contralateral hemisphere; yet, both hemispheres are involved when acquiring new bimanual skills, albeit different regions for different functionalities (27, 28). According to the dynamic dominance hypothesis, each hemisphere is responsible for different aspects of task performance and optimizes different costs (29). In this framework, the control of the dominant limb is based on a feed-forward control mechanism to anticipate the dynamics of the task. In contrast, control of the nondominant limb is more reactive and relies on sensory feedback to achieve higher positional accuracy (30) and is thus better suited for maintaining stability during the task (31–35). Hence, human hand selection seems to be affected by both performance asymmetries and task demands (36).

Another view is provided by Johansson et al. (37), who suggest that the brain assigns differentiated roles to the two hands according to spatial relations between the forces that each hand needs to produce to achieve the task goal. Hence, role assignment is no longer solely determined by handedness nor is it fixed. Instead, the two hands can switch roles as the task evolves. Such an active control of role assignment may be more effective in a large variety of tasks. For example, switching roles across hands could prevent having to reorient or regrasp the object. This is particularly useful when the external task conditions change (38).

Taxonomies and Virtual Fingers

Despite the large number of degrees of freedom in hand movements, in daily life, humans tend to use a subset of the possible ways they can shape their hands. To encapsulate these typical hand movements, hand postures have been categorized according to the number of used fingers and their positioning (e.g., precision vs. power grasp) and the relation among the used fingers (e.g., relative position of thumb with respect to other fingers). There are three main methods to taxonomize the human grasp (39–42): 1) hand-centric and motion-centric manipulation taxonomies (43); 2) the haptic action-focused taxonomy for disassembly tasks (44); and 3) taxonomies to simplify control of robotic hands for prosthetics and rehabilitation (45). Although all these taxonomies are powerful and representative of a large set of regular hand poses, they only focus on common unimanual tasks and do not account for the combinations of hand poses in bimanual tasks. They also have not considered more rare hand poses adopted in special crafts, such as watchmaking, which is discussed in this paper.

One promising explanation to hand and finger shaping is the concept of virtual finger (VF), introduced by Arbib, Iberall, and colleagues (46–48). Instead of focusing on the abundant mechanical degrees of freedom of individual fingers, the virtual finger concept analyzes hand poses at a functional level. Each virtual finger represents one unit of the hand aimed to achieve an independent function, for instance, applying force in a desired direction. One virtual finger may consist of one or more real fingers or even part of the hand; components of one virtual finger move together to contribute to the same function in the task. For instance, to grasp and lift a bottle of water, the thumb and other fingers are usually placed on the opposite side of the bottle to apply forces in opposition; fingers that press the bottle against the thumb can be considered as one virtual finger. It was suggested that control commands are directly sent from the brain to the virtual finger, and the feedback is also received from the virtual finger instead of the real fingers (49). Such an approach explains how the brain may simplify the control of the complex hand by reducing the number of degrees of freedom to match the task’s demands. The VF concept was supported through experimental evidence in a series of studies of grasping (50–52). In this study, we use the VF concept to describe the positioning of the fingers on the tool in relation to different task demands.

Hypotheses and Task Considered

This study seeks to understand how different task demands affect the hand pose selection for each hand and the role assignment across the two hands. To this end, we chose a task that requires precise control of position, orientation, and force of the tool. Specifically, we selected a precision screwing task that is taken from the behavioral repertoire of Swiss watchmakers, a profession that requires years of training. Screwing is likely the most common task in watchmaking. During both positioning and insertion of the piece, the two hands move in coordination through fast and precise movements. Positioning of the watch face and control of force are crucial to avoid breaking the fragile elements of the watch.

To reduce the problem and make it accessible to experimental study, we selected one watch face and one screw used in the first year of watchmaking training at the École Technique de la Vallée de Joux (a 150-yr-old school in watchmaking in Switzerland). The screw was not only small enough to require the desired precision and bimanual coordination but also large enough to be amenable to our subjects, who were knowledgeable in screwing small elements but were not professional watchmakers.

We hypothesized that hand pose selection and role distribution across the two hands is strongly influenced by task demands. Task demands were manipulated by creating restrictions on the manipulated object in two experimental conditions. We assume that the more degrees of freedom required to be controlled by the task, the higher the task demands. In the fixed-base condition, the degrees of freedom of the task were reduced by attaching the watch to the table. In the free-base condition, the watch was free to slide on the table, forcing simultaneous control of both watch and screw, thus corresponding to higher task demands.

Specifically, we hypothesized that the number of degrees of freedom that need to be controlled in the task will affect the choice of hand poses across the two hands. We expected that less constrained tasks, i.e., tasks with fewer degrees of freedom to control for, would lead to more variation in the selected combinations of hand poses. Conversely, tasks with more degrees of freedom to control for, and for which fewer solutions exist, would lead to less variation in the hand pose combinations and would increase the influence of hand dominance on role distribution.

MATERIALS AND METHODS

Participants

Twenty participants (age: 24.2 ± 6.0 yr, 5 women) were recruited from engineering students at École Polytechnique Fédérale de Lausanne and from the École Technique de la Vallée de Joux. All subjects had experience in using precision screwdrivers, as per their training, but were not proficient at watchmaking. All subjects were right-hand dominant, as assessed by the Edinburgh handedness inventory (53). They were divided into two groups with 10 subjects each, performing one of the two experimental conditions. Before the experiment, subjects were informed about the experimental procedures and gave their written informed consent. The study was approved by the Institutional Ethics Review Board at the École Polytechnique Fédérale de Lausanne.

Apparatus

An original watch face (type ETA 17 JEWELS 5494, diameter 36.6 mm) and a flat screwdriver (length: 89.0 mm, diameter of tip: 1.6 mm) served as experimental apparatus and tool (Fig. 1A). The screwdriver is an original jewelers’ precision screwdriver. One GoPro camera (GoPro Inc.) was placed on the table ∼20 cm in front of the subjects to record hand and finger movements at 60 Hz. The camera was mounted so that it could capture a clear front view of the hands and all finger movements.

Figure 1.

Experimental setup and apparatus. A: one GoPro camera was placed in front of the subjects to record hand movements while they performed the task. B: original watch face as used for the experiment. In the ex post facto experiments for analyzing hand poses, an ATI F/T (Force/Torque) sensor was mounted underneath the watch face to measure the applied resultant force and torque.

The experimenter conducted an ex post facto study to analyze the forces and torques generated by different hand poses. To this purpose, an ATI Nano17 miniature force/torque (F/T) sensor (ATI Industrial Automation, Inc.; Fig. 1B, resolution: 1/80 N for force sensing and 1/16 Nmm for torque sensing) was mounted underneath the watch face using a three-dimensional printed support. The multiaxis sensor measured the forces and torques applied on the watch screw in all three Cartesian coordinates (X, Y, and Z), using as reference the horizontal plane on which the watch face rested (Fig. 1B). The sensor captured the torques and forces generated along the Z-axis. An anticlockwise movement of the screwdriver generated torque in the Z+ direction (the vector in red), and pressing the screwdriver cap led to a force in the Z− direction (the vector in green), opposing the torque’s direction. Note that the sensor could only capture the resultant force and torque applied on the watch face; thus, the force or torque controlled by each individual hand could not be inferred from the resultant sensor recording. Therefore, subjects performed the task without this sensor in the experiment. The sensor was only used by the experimenter in the ex post facto study for analyzing each one of the observed single hand poses. Analysis of recorded subjects’ motion enabled the experimenter to mimic the hand poses adopted by the subjects.

Experimental Task

Subjects sat on a chair facing a table with the watch face laid flat on the desk surface (Fig. 1A). They were instructed to use a slot-head screwdriver to dismount a tiny screw (Fig. 1B, head diameter: 1.6 mm, total length: 3.1 mm) on the watch face. No specific instructions were given on how they should do this. Regardless of how they completed the task, there were always two steps: 1) the localization step, in which the subject picked up the screwdriver and inserted its tip onto the screwhead’s slot; 2) the execution step, in which the subject maneuvered the screwdriver to dismount the screw. Subjects had to rotate the screwdriver in an anticlockwise direction and generate torque in the vertical (+Z) direction (Fig. 1B). During this process, a constant downward pressure had to be maintained on the head of the screwdriver to avoid cam-outs. Cam-outs were a common type of failure, where the screwdriver slid out of the groove of the screwhead during rotation (54). It slowed down the task considerably and could even break the tiny watch components. Two experimental conditions were contrasted.

Fixed-base condition.

One group of subjects (10 subjects) performed with the watch face immobilized on the table’s surface (Fig. 2A). In this scenario, subjects only needed to control the six degrees of freedom of the screwdriver and did not need to be concerned about potential movements of the watch face while dismounting the screw.

Figure 2.

Typical hand pose combinations observed in each experimental condition. A: in the fixed-base condition, subjects displayed more diversity in their hand poses. 1: in most cases, one hand (right hand) rotated the screwdriver, and the other hand (left hand) assisted manipulation by stabilizing the screwdriver. 2: in other cases, the fingers of the two hands moved in coordination, while the right finger provided stabilization. 3: in a few trials, subjects used only one hand. B: in the free-base condition the dominant (right) hand typically controlled the screwdriver and the nondominant (left) hand stabilized the watch face.

Free-base condition.

Another group of subjects (10 subjects) performed the task where the watch face was free to move on the table surface (Fig. 2B). To complete the task, subjects had to prevent potential movements of the watch face while dismounting the screw. This task required the control of not only the six degrees of freedom of the screwdriver but also three additional degrees of freedom for the translational and the rotational movement of the watch face.

In both conditions, subjects were free to place their hands as they wished using either one hand or both hands. They had sufficient time to get familiar with the screwdriver before starting the experiment. Before each recording, subjects first mounted the watch screw tightly into its anchor. This part of the action was not monitored and subjects could take as much time as they wished. The experimenter made sure that the screw was tightly mounted and then the subject started to dismount the screw. Each subject performed five consecutive trials in total. They were encouraged to complete each trial as fast and as accurately as possible. If a cam-out happened, the subject stopped, corrected the failure, and then resumed the task. Task performance was evaluated based on the number of cam-outs and the average rotation time of the screwdriver. A tightly mounted screw needed to be rotated about five complete rounds until it could be dismounted.

Data Analysis

Video analysis and dependent measures.

Subjects were video-recorded to determine failure rates, time of finger rotation, and finger positioning on the screwdriver. Note that for the full experiment, only video analysis was available. In the analysis, the continuous video recordings were first segmented into trials, with each segment containing one complete unscrewing manipulation. The start of each trial was defined when the subject picked up the screwdriver. The trial ended when the subject had successfully unmounted the screw and the screw was completely out of its anchor. The segmentation error was maximally ±3 frames, equivalent to 50 ms (video was recorded at 60 Hz). This accuracy was the experimenter’s reported uncertainty in deciding the starting/ending frame of movements.

For each trial, five types of information were extracted. 1) t_s, the start time when the subject had successfully localized and inserted the tip of the screwdriver into the screwhead and was about to rotate the screwdriver. 2) t_e, the end time when the subject stopped rotating the screwdriver completing the task. 3) N_m, the number of rhythmic finger movements. Each cycle started with finger flexion when the finger contacted the screwdriver and was about to rotate; the movement was completed when the finger extended and stretched back to its original position. 4) N_f, the number of cam-outs or failures that occurred during the trial. 5) The combination of left and right hand poses used for the task, summarized in the hand pose matrix H (see Hand pose taxonomy).

Three raters independently annotated the video recordings. Two raters had no experience in the study and were naive to the experimental task and one rater was the experimenter. All raters labeled the hand poses according to the hand pose taxonomy described in Hand pose taxonomy. The results demonstrate a 98% match rate among the three raters; only 2% of hand poses were classified into different categories. This occurred when subjects added or removed one active finger during manipulation.

Analysis of task performance.

To evaluate the subjects’ task performance, two quantitative metrics were defined: 1) failure rate, δ, and 2) movement time, t.

Performance metrics.

failure rate.

The average number of cam-outs across trials, δ, served as a descriptive measure for the general difficulty and performance of the task. The number of cam-outs could be larger than the number of trials, as subjects could fail more than once before completing a task trial.

movement time.

The average finger movement time for each trial, t, was calculated as the total trial time divided by the number of rhythmic finger movements N_m per trial. One complete finger rotation included one flexion and one extension of the finger. In case of a cam-out, only the motion segment before the cam-out was used to calculate t of this trial.

Hand pose taxonomy.

Virtual finger analysis.

For each hand pose used by subjects, its function was analyzed by mapping virtual fingers (VF) onto the real fingers. Anatomically, one VF may involve one or more real fingers, or even including the palm. Each independent function, e.g., applying force in one direction, was represented by one virtual finger. Therefore, one hand pose could involve several VFs. Following Arbib, Iberall, and colleagues (46–48), the components within a virtual finger are controlled as one unit.

For example, in the typical hand pose combination in the free-base condition (Fig. 3A1), the function of the subject’s left hand (Fig. 3A2) was achieved by two VFs that constrained the movement of the watch face. Each VF contained one real finger: VF1, thumb and VF2, index finger, both shown in blue dashed lines. The function of the subject’s right hand (Fig. 3A3) was mapped to three VFs: the index finger (VF1, green color) maintained the force in the vertical direction; the middle finger and the ring finger constituted VF2, since they moved in unison to apply force in the same direction on the screwdriver; the thumb was mapped to VF3 (both VF2 and VF3 are in red color), as it moved opposite to VF2 and exerted an opposing force. VF2 and VF3 moved in synergistic coordination and applied opposing forces to rotate the screwdriver. Using this hand pose combination, both task-demanded force and task-demanded torque could be controlled by the three VFs in the right hand.

Figure 3.

Two typical combinations of hand poses observed in the free-base condition (A) and the fixed-base condition (B). In each case, 1 shows the observed typical hand pose combination, 2 and 3 illustrate the individual hand pose as well as the task demands controlled by each hand.

In the fixed-base condition in Fig. 3B1, the subject used a different manipulation strategy: the subject assigned the left index finger to one VF for maintaining the pressure (Fig. 3B2). With the right hand (Fig. 3B3), the subject used two VFs: the thumb was assigned to VF1 and index and middle fingers were assigned to VF2, since both index and middle fingers were in contact with the screwdriver to apply force in the same direction. These two VFs of the right hand applied forces in opposing directions to rotate the screwdriver, whereas the task-demanded force could be applied by the VF on the left hand. This analysis of virtual finger assignment was conducted for all observed hand poses, and the results are summarized in Fig. 4.

Figure 4.

Hand pose taxonomy. For each hand pose, the number of virtual fingers (#VF) could be 1, 2, or 3. Fingers are represented by numeric values (1: thumb, 2: index finger; 3: middle finger, 4: ring finger, and 5: little finger) and letter P for palm. The number of real fingers used to complete the task was registered as the number of active fingers (#AF). Vectors in the task coordinate frame (F/T Control) represent the force or torque that could be generated by manipulating the tool using the corresponding hand pose. The solid and dashed vectors denote continuous and discontinuous maintenance of force or torque. The light blue ellipsoid indicates the region of potential disturbances. Poses 1, 7, 8, 9, and 10 are specific to tasks in this study and have not been reported previously.

Hand pose taxonomy.

Figure 4 summarizes the hand poses adopted by either the dominant or nondominant hand in both task conditions, both during localization of the tool and execution of the task. The hand poses are ranked according to the number of virtual fingers (#VF) and the number of active real fingers (#AF) involved. Hand pose 1 is task specific and has not been categorized in the literature. Hand poses 2, 3, 4, 5, and 6 correspond to hand poses reported previously in the GRASP taxonomy by Feix et al. (41): inferior pincer (type 33), palmar pinch (type 9), prismatic two-finger (type 8), prismatic three-finger (type 7), and prismatic four-finger (type 6). Pose 7 is adjusted from the stick type (type 29), whereas poses 8, 9, and 10 were also newly observed in this study. These new poses can be considered as integrations of the prismatic grasp poses (type 9, 8, and 7) and the index finger extension pose (type 17). Notice that although the right hand is shown for illustration in most cases in Fig. 4, all hand poses could be adopted by either hand, regardless of handedness.

Measuring forces and torques of hand poses.

The experimenter replicated each of the observed hand poses summarized in Fig. 4 with an ATI F/T sensor mounted underneath the watch face for an ex post facto experiment. For each hand pose, the force/torque recorded by the experimenter is summarized in Fig. 4 (bottom row, F/T Control). We used dimensionless unit vectors to represent the force (in green color) or torque (in red color) that could be applied in the directions of interest (force in the Z+ direction and torque in the Z− direction). Force and torque components that were likely to occur inside the X-Y plane were perpendicular to the directions of interest; thus, they were considered disturbances. We used a dimensionless ellipse (in light blue color) on the X-Y plane to indicate the existence of these potential disturbances.

Hand pose 1 could only apply constant force by exerting axial pressure onto the screwdriver. Hand pose 2 was the only pose used to manipulate the watch face. It stabilized the watch face but could not control any task demands. Hand poses 3–6 included two VFs that moved in coordination to generate the torque needed to rotate the screwdriver. Hand poses 7–10 controlled both force and torque simultaneously. They generated and maintained continuous force on the screwdriver cap in the Z− direction with either the palm (pose 7) or the index finger (poses 8 –10). Torque could only be generated in a discontinuous manner, as fingers that rotated the screwdriver had to move back and forth, and the switch of finger movements interrupted the torque production.

Figure 5 summarizes the hand pose combinations that were observed in both experimental steps and in both experimental conditions. Each individual hand in the combination was categorized according to Fig. 4. Hand poses with the same number of virtual fingers were considered to be in the same category in Fig. 5, regardless of the number of active fingers. For example, hand pose combinations (0, 3), (0, 4), and (0, 5) were categorized in the same combination in Fig. 5 (first column), although they had different numbers of active fingers.

Figure 5.

Combinations of hand poses involving both hands that were observed in the two experimental conditions. They are categorized according to their virtual fingers analyzed in Fig. 4. For each hand pose (left or right), #VF indicates the number of virtual fingers in this pose and F/T Control illustrates the task condition (force, torque, or the motion of the base) being controlled.

Hand pose matrix.

The hand pose matrix H was defined to summarize the hand pose combinations observed in each experimental condition (free-base and fixed-base) and each experimental step (localization and execution). First and second index of each matrix entry indicate the left- and right-hand pose categories (0–10, according to the taxonomy summarized in Fig. 4, 0 for unused hand) of this combination. Each entry’s value represents the total number of trials that this hand pose combination was observed, separately for the two steps and the two experimental conditions. Four H matrices distinguished between the localization and execution steps in the free-base and fixed-base conditions: $H_{\mathrm{f}\mathrm{r}\mathrm{e}\mathrm{e}}^{\mathrm{l}\mathrm{o}\mathrm{c}}$, $H_{\mathrm{f}\mathrm{r}\mathrm{e}\mathrm{e}}^{\mathrm{e}\mathrm{x}\mathrm{e}}$, $H_{\mathrm{f}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{d}}^{\mathrm{l}\mathrm{o}\mathrm{c}}$, and $H_{\mathrm{f}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{d}}^{\mathrm{e}\mathrm{x}\mathrm{e}}$. For example, $H_{\mathrm{f}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{d}}^{\mathrm{e}\mathrm{x}\mathrm{e}}$(2, 7) = 3 indicates that the hand pose combination of left pose 2 and right pose 7 was observed in the execution step in three trials of the fixed-base condition.

The hand pose matrix provided a comprehensive overview of all observed hand poses, as well as a visualization of functional distributions across both hands. A symmetric role assignment between hands was reflected in the entries along the diagonal of H, when both hands used identical poses to operate the tool. Role assignment was asymmetric, when both hands used different poses. A right-hand-lead role assignment was reflected by entries in the upper triangular region of the matrix. Conversely, entries located in the lower triangular region corresponded to a left-hand-lead role assignment. Along the diagonal of the matrix, the total number of active fingers in both hands increases from the top left corner (both hands were unused) to the bottom right corner (all fingers from both hands were used). Moreover, unimanual manipulation trials corresponded to the first row (left hand unused) and the first column (right hand unused) of H.

Entropy and structural similarity index.

To quantify how sparse the matrix H was, the entropy value, h, was calculated (55):

$$h=-\sum _{i=1}^n{p}_i\mathrm{l}\mathrm{o}\mathrm{g}\left(p_i\right),$$

where n is the number of observed different hand pose combinations, which corresponds to the number of H’s nonzero entries; p_i denotes the probability of the ith hand pose combination observed in the experiment. A high value of h indicated the task was completed using multiple feasible hand pose combinations. The notation of h follows the same convention as H. For example, the entropy of hand pose matrix $H_{\mathrm{f}\mathrm{r}\mathrm{e}\mathrm{e}}^{\mathrm{l}\mathrm{o}\mathrm{c}}$ is denoted as $h_{\mathrm{f}\mathrm{r}\mathrm{e}\mathrm{e}}^{\mathrm{l}\mathrm{o}\mathrm{c}}$.

The structural similarity index (SSIM) was originally developed to compare the similarity between two images (56). We applied this index to measure structural similarity between two matrices H_X and H_Y, calculated as:

$$s\left(H_X,H_Y\right)=\frac{\left(2{\mathrm{\mu }}_X{\mathrm{\mu }}_Y+C_1\right)\left(2{\mathrm{\sigma }}_{XY}+C_2\right)}{\left({\mathrm{\mu }}_X^2+{\mathrm{\mu }}_Y^2+C_1\right)\left({\mathrm{\sigma }}_X^2+{\mathrm{\sigma }}_Y^2+C_2\right)}, s\in \left[-1\text{,1}\right],$$

where μ_X and μ_Y are local means of H_X and H_Y, respectively; σ_X and σ_Y are the standard deviations of H_X and H_Y, respectively; σ_XY is the cross-covariance of H_X and H_Y. C₁ and C₂ are regulation constants that depend on the value range of the matrix. They were used to stabilize the division when μ or σ were close to zero. The value of s(H_X, H_Y) ranges from −1 (disparate structures) to +1 (identical structures).

Statistical analysis.

The relation between task performance metrics (failure rate δ and movement time t) were fitted with linear regression. All data were analyzed using MATLAB. A one-way ANOVA was used to evaluate statistical significance in the comparison of the task performance between groups.

RESULTS

Speed and Accuracy of Performance

Each subject spent around 10 min participating in the experiment, including some time for getting familiar with manipulation tools. Subjects were required to repeat the experimental task for only five trials to avoid fatigue of hand and fingers in this intense movement. In each trial, subjects spent around 4.9–9.2 s to complete the rotations (6.4 s on average).

In the free-base condition, the subjects’ average finger movement time was t_free = 0.50 ± 0.08 s (range: 0.39–0.67 s). The subjects’ overall failure rate was δ_free = 0.08 ± 0.14 (range: 0–0.40). Figure 6 illustrates three examples of cam-outs. The average movement time was negatively correlated with the average number of cam-outs for all 10 subjects with a correlation coefficient ρ(t_free, δ_free) = −0.68, suggesting failures were more likely to occur in faster movements.

Figure 6.

Hand pose combinations observed when failures happen during the execution step under the fixed-base condition. A: $H_{\mathrm{f}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{d}}^{\mathrm{e}\mathrm{x}\mathrm{e}}$(1, 3). B: $H_{\mathrm{f}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{d}}^{\mathrm{e}\mathrm{x}\mathrm{e}}$(4, 1). C: $H_{\mathrm{f}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{d}}^{\mathrm{e}\mathrm{x}\mathrm{e}}$(4, 8).

In the fixed-base condition, the subjects’ average finger movement time was t_fixed = 0.36 ± 0.07 s (range: 0.27–0.49 s). The subjects’ overall failure rate was δ_fixed = 0.12 ± 0.17 (range: 0–0.40). There was a negative correlation between average movement time and average failure rate for all 10 subjects with ρ(t_fixed, δ_fixed) = −0.66.

Subjects performed the task significantly faster in the fixed-base condition compared with that in the free-base condition. The difference in the average movement times was significantly different from zero (F_1,18 =17.24, P < 0.001). However, the subjects’ overall failure rates were not significantly different in these two conditions (F_1,18 =0.33, P = 0.57).

Hand Pose Analysis

The hand poses in each step of the task (localization and execution) were identified to analyze how hand pose selection was influenced by the number of degrees of freedom that needed to be controlled.

Hand poses during localization.

The matrices $H_{\mathrm{f}\mathrm{r}\mathrm{e}\mathrm{e}}^{\mathrm{l}\mathrm{o}\mathrm{c}}$and $H_{\mathrm{f}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{d}}^{\mathrm{l}\mathrm{o}\mathrm{c}}$ summarized the hand pose combinations observed during the localization step, in both free-base and fixed-base conditions, respectively.

Free-base condition.

In the free-base condition, subjects consistently used the hand pose combinations, represented as matrix entries $H_{\mathrm{f}\mathrm{r}\mathrm{e}\mathrm{e}}^{\mathrm{l}\mathrm{o}\mathrm{c}}$(2, 8), $H_{\mathrm{f}\mathrm{r}\mathrm{e}\mathrm{e}}^{\mathrm{l}\mathrm{o}\mathrm{c}}$(2, 9), and $H_{\mathrm{f}\mathrm{r}\mathrm{e}\mathrm{e}}^{\mathrm{l}\mathrm{o}\mathrm{c}}$(2, 10), respectively (Fig. 7A). Hand pose 2 is used by the left hand to stabilize the watch face on the table by using the thumb and index finger to pinch the watch face along its circumference. This pose is combined with the right hand in a prismatic pose with the index finger pressing in all trials. Among these trials, the right hand’s number of active fingers varies from 3 (pose 8, only the thumb and middle finger pinched the screwdriver, accounted for 2%) to 5 (pose 10, the thumb pressed against all the remaining active fingers, accounted for 52%). In the remaining 46% trials, the right hand was in pose 9 and the thumb pressed against the middle and ring fingers.

Figure 7.

Hand pose matrix and corresponding composition by subjects in the localization step of the free-base condition. A: hand pose matrix of the free-base condition $H_{\mathrm{f}\mathrm{r}\mathrm{e}\mathrm{e}}^{\mathrm{l}\mathrm{o}\mathrm{c}}$ visualized as a heat map with the color bar indicating the magnitude of values. Matrix entries are the number of observations. Empty cells indicate no observations. Hand pose combinations in the small figure correspond to one of the categorized combinations in Fig. 5. B: number of observations split by subjects. The X-axis lists the three hand pose combinations observed in this step and condition. The Y-axis represents the number of observations for each hand pose combination. The colors indicate each subject’s contribution.

Concurrently, the index finger of the right hand in pose 8, 9, or 10 pressed the screwdriver head to constrain its movement in the axial direction during insertion, while the remaining active fingers of the right hand, i.e., the thumb, middle and ring finger (including the little finger in pose 10) were in contact with the side of the screwdriver to stabilize it. Poses 9 and 10 were used the most across all trials. The palmar pinch pose (pose 8) was used only once. Despite the differences in the number of active fingers, the right-hand poses 8, 9, and 10 shared the same number of virtual fingers: the thumb and the index finger each served as one virtual finger, whereas the remaining active fingers moved as a group to form the third virtual finger.

The variability of the observed hand pose combinations during localization in the free-base condition was quantified by the entropy metric. The very small number $h_{\mathrm{f}\mathrm{r}\mathrm{e}\mathrm{e}}^{\mathrm{l}\mathrm{o}\mathrm{c}}$= 0.17 indicated the high concentration of hand pose combinations (Fig. 7A). The bar chart in Fig. 7B splits the observations by subject and illustrates the hand pose combinations used for localization. The height of each bar represents the number of trials. Four subjects changed their hand poses across trials (S4, S7, S9, and S10), whereas the remaining subjects used the identical hand pose combination throughout all trials.

Fixed-base condition.

Figure 8A shows the hand pose matrix for localizing screwdriver in the fixed-base condition. Subjects in up to 32% of trials used the prismatic grasping pose combination $H_{\mathrm{f}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{d}}^{\mathrm{l}\mathrm{o}\mathrm{c}}$(4, 9) to operate the screwdriver. The left hand used pose 4 to pinch the tool involving the thumb, index, and middle fingers. Simultaneously, the right hand having more active fingers and virtual fingers adopted pose 9 to pinch and press the screwdriver. In this pose, the index finger pressed on the screwdriver head to restrict its movement along the axial direction when it was inserted into the screw slot, while the thumb, middle, and ring fingers pinched the screwdriver to control the position of the tip. In a few trials, subjects used hand pose combinations similar to (4, 9), although with slight adjustments captured in the matrices as $H_{\mathrm{f}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{d}}^{\mathrm{l}\mathrm{o}\mathrm{c}}$(4, 8), $H_{\mathrm{f}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{d}}^{\mathrm{l}\mathrm{o}\mathrm{c}}$(5, 9), and $H_{\mathrm{f}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{d}}^{\mathrm{l}\mathrm{o}\mathrm{c}}$(4,10). These adjustments included adding or removing active fingers to form variations in the prismatic poses.

Figure 8.

Hand pose matrix and the corresponding composition by subject in the localization step of the fixed-base condition. A: hand pose matrix of the free-base condition $H_{\mathrm{f}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{d}}^{\mathrm{l}\mathrm{o}\mathrm{c}}$ visualized as a heat map with the color bar indicating the magnitude of values. Matrix entries are the number of observations. Empty cells indicate no observations. Hand pose combinations in the small figure correspond to one of the categorized combinations in Fig. 5. B: number of observations split by subjects. The X-axis lists all the hand pose combinations observed in this step and experimental condition. The Y-axis represents the number of observations for each hand pose combination. The colors indicate each subject’s contribution.

In contrast, 28% of localization hand pose combinations used fewer active fingers and also fewer virtual fingers, denoted in the matrix as $H_{\mathrm{f}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{d}}^{\mathrm{l}\mathrm{o}\mathrm{c}}$(0–1, 3–5). In such combinations, the left hand was either not used [pose 0, in $H_{\mathrm{f}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{d}}^{\mathrm{l}\mathrm{o}\mathrm{c}}$(0, 4–5)] or used the index finger (or both thumb and index finger) to press the screwdriver head [pose 1, in $H_{\mathrm{f}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{d}}^{\mathrm{l}\mathrm{o}\mathrm{c}}$(1, 3–5)]. This achieves the same functionality as the index finger of the right hand [pose 9, in $H_{\mathrm{f}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{d}}^{\mathrm{l}\mathrm{o}\mathrm{c}}$(4, 9)], as discussed under Free-base condition. The right hand in pose 3, 4, or 5 simply pinched the screwdriver without pressing its head. Such combinations resulted in an equivalent virtual finger assignment across hands as hand poses 8, 9, and 10 that were commonly used in the free-base condition [see $H_{\mathrm{f}\mathrm{r}\mathrm{e}\mathrm{e}}^{\mathrm{l}\mathrm{o}\mathrm{c}}$(2, 8–10)]. These hand pose combinations had one virtual finger pressing the screwdriver and multiple virtual fingers for holding the screwdriver.

Figure 8B illustrates the observed hand pose combinations split by subjects. The choice of hand pose combinations visibly varied more than in the free-base condition. Seven out of 10 subjects used different hand pose combinations across the trials (S1, S2, S3, S5, S6, S8, and S9). The higher variability of hand pose combination in the fixed-base condition was quantified by the higher entropy value $h_{\mathrm{f}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{d}}^{\mathrm{l}\mathrm{o}\mathrm{c}}$= 0.52.

Hand poses during execution.

The matrices $H_{\mathrm{f}\mathrm{r}\mathrm{e}\mathrm{e}}^{\mathrm{e}\mathrm{x}\mathrm{e}}$and $H_{\mathrm{f}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{d}}^{\mathrm{e}\mathrm{x}\mathrm{e}}$ summarize the hand pose combinations observed during the execution step, in both the free-base and fixed-base conditions, respectively.

Free-base condition.

For the execution step, there was high consistency in the choice of hand poses both across subjects and trials, as is clear from the matrix in Fig. 9A. Subjects used the hand pose combination $H_{\mathrm{f}\mathrm{r}\mathrm{e}\mathrm{e}}^{\mathrm{e}\mathrm{x}\mathrm{e}}$(2, 8) in 48% trials and $H_{\mathrm{f}\mathrm{r}\mathrm{e}\mathrm{e}}^{\mathrm{e}\mathrm{x}\mathrm{e}}$(2, 9) in 52% trials. These hand poses were similar to those during localization of the tool (see Fig. 7A) and revealed a clear predominance of the right hand, concentrated in the upper triangular region of the matrix. In most trials, the right hand formed a prismatic pose, where the index finger pressed the screwdriver head to maintain a downward force when rotating the screwdriver. Simultaneously, the left hand was in the inferior pincer pose to stabilize the watch face along the circumference. The predominance of these two hand pose combinations is expressed by the small entropy value $h_{\mathrm{f}\mathrm{r}\mathrm{e}\mathrm{e}}^{\mathrm{e}\mathrm{x}\mathrm{e}}$= 0.12.

Figure 9.

Hand pose matrix and corresponding composition by subject in the execution step of the free-base condition. A: hand pose matrix of the free-base condition $H_{\mathrm{f}\mathrm{r}\mathrm{e}\mathrm{e}}^{\mathrm{e}\mathrm{x}\mathrm{e}}$ visualized as a heat map with the color bar indicating the magnitude of values. Matrix entries are the number of observations. Empty cells indicate no observations. Hand pose combinations in the small figure correspond to one of the categorized combinations in Fig. 5. B: number of observations split by subjects. The X-axis lists all the hand pose combinations observed in this step and experimental condition. The Y-axis represents the number of observations for each hand pose combination. The colors indicate each subject’s contribution.

This is also visible in the distribution of hand poses across subjects and trials (Fig. 9B). Subjects seldom changed hand poses: six subjects (S1, S4, S6, S7, S8, and S9) used identical pose combinations in all trials; three subjects (S2, S3, and S5) changed only once, and one subject (S10) used $H_{\mathrm{f}\mathrm{r}\mathrm{e}\mathrm{e}}^{\mathrm{e}\mathrm{x}\mathrm{e}}$(2, 9) twice and $H_{\mathrm{f}\mathrm{r}\mathrm{e}\mathrm{e}}^{\mathrm{e}\mathrm{x}\mathrm{e}}$(2, 8) three times.

Task performances quantified by failure rate (accuracy) and movement time (speed) for each hand pose combination are illustrated, respectively, in Fig. 10, A and B. The two observed hand pose combinations (Fig. 9A) both correspond to an 8% failure rate (Fig. 10A) with a similar average movement time of 0.51 s and 0.49 s (Fig. 10B), respectively.

Figure 10.

Task performance associated with each execution hand pose combination. A: average failure rate. B: average finger(s) movement time. The shade of the color bar is proportional to the values.

One-way ANOVA analysis of movement time indicated that the subjects who changed hand poses across trials performed the task faster than those who consistently used one hand pose combination (F_1,48 = 8.54, P = 0.005). However, the average finger movement time was 0.54 ± 0.15 s (mean ± SD) before changing hand poses and 0.54 ± 0.10 s after changing hand poses, across all trials. Statistically, there is no significant difference revealed (F_1,10 = 0.01, P = 0.92). Moreover, all subjects continued using the identical hand poses after failed trials, and the average failure rate remains on the similar level after changing hand poses (F_1,48 = 2.95, P = 0.09). Therefore, the task performance in both aspects are not related to the change of hand poses.

Fixed-base condition.

In contrast to the free-base condition, hand poses in the fixed-base condition revealed more variation in their role distribution. This is best visualized by the execution hand pose matrix $H_{\mathrm{f}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{d}}^{\mathrm{e}\mathrm{x}\mathrm{e}}$, where the entries are quite dispersed (Fig. 11A). This was quantified by the entropy metric $h_{\mathrm{f}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{d}}^{\mathrm{e}\mathrm{x}\mathrm{e}}$ = 0.41. Inspection of the distribution of hand poses across subjects (Fig. 11B) shows that most subjects changed their hand pose combinations between trials. Only three subjects (S2, S7, and S10) used the same hand pose combination across all trials. The most frequently observed hand pose combinations were $H_{\mathrm{f}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{d}}^{\mathrm{e}\mathrm{x}\mathrm{e}}$(4–5, 1), used in 20 trials, corresponding to a 40% occurrence rate. The entries are located in the lower-triangular region of the matrix (Fig. 11A). In these trials, the right hand was in charge of maintaining the downward force on the screwdriver. Simultaneously, the left hand adopted the prismatic pose to rotate the screwdriver with the thumb, the index finger, and/or the middle finger.

Figure 11.

Hand pose matrix and the corresponding subjects composition in the execution step of the fixed-base condition. A: hand pose matrix of the free-base condition $H_{\mathrm{f}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{d}}^{\mathrm{e}\mathrm{x}\mathrm{e}}$ visualized as a heat map with the color bar indicating the magnitude of values. Matrix entries are the number of observations. Empty cells indicate no observations. Hand pose combinations in the small figure correspond to one of the categorized combinations in Fig. 5. B: number of observations split by subjects. The X-axis lists all the hand pose combinations observed in this step and experimental condition. The Y-axis represents the number of observations for each hand pose combination. The colors indicate each subject’s contribution.

In 30 out of 50 total trials, the right hand rotated the screwdriver, shown in all matrix entries in the upper-triangular region of $H_{\mathrm{f}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{d}}^{\mathrm{e}\mathrm{x}\mathrm{e}}$, whereas in the remaining 20 trials, the left hand took this role shown by entries in the lower-triangular region of $H_{\mathrm{f}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{d}}^{\mathrm{e}\mathrm{x}\mathrm{e}}$ (Fig. 11A).

Task performances associated with each hand pose combination in the fixed-base condition are summarized in Fig. 12, A and B, respectively. The combination $H_{\mathrm{f}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{d}}^{\mathrm{e}\mathrm{x}\mathrm{e}}$(1, 3) had the highest failure rate of 40% (Fig. 12A). This occurred when the right hand used only two active fingers (the thumb and the index finger) to rotate the screwdriver. This hand pose is associated with the shortest average movement time of 0.27 s (Fig. 12B). Notably, the failure rate decreased to 18% if the left hand served as the lead role to execute a similar pose, denoted by $H_{\mathrm{f}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{d}}^{\mathrm{e}\mathrm{x}\mathrm{e}}$(4, 1) (Fig. 12A). This combination had an average movement time of 0.33 s (Fig. 12B). The hand pose combination $H_{\mathrm{f}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{d}}^{\mathrm{e}\mathrm{x}\mathrm{e}}$(4, 8) was associated with a failure rate of 20%. Using this combination, fingers from both hands alternately rotated the screwdriver with the right index finger maintaining downward pressure on the screwdriver head during rotation. Hand pose combinations with a zero failure rate were located in the upper-triangular region of the matrix $H_{\mathrm{f}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{d}}^{\mathrm{e}\mathrm{x}\mathrm{e}}$ (Fig. 12A). They were associated with the relative longer finger(s) movement time (Fig. 12B).

Figure 12.

Task performance associated with each execution hand pose combination. A: average failure rate. B: average movement time. The shade of the color bar is proportional to the values.

The change of hand poses does not seem to be caused by failure, as only one subject (S4) changed hand pose from $H_{\mathrm{f}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{d}}^{\mathrm{e}\mathrm{x}\mathrm{e}}$(4, 8) to $H_{\mathrm{f}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{d}}^{\mathrm{e}\mathrm{x}\mathrm{e}}$(4, 1) once after a cam-out, and then used $H_{\mathrm{f}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{d}}^{\mathrm{e}\mathrm{x}\mathrm{e}}$(4, 1) in the remaining trials. The average finger movement time was 0.37 ± 0.10 s (mean ± SD) across all subjects before changing hand poses and 0.38 ± 0.09 s after. Statistics did not reveal any significant differences (F_1,20 = 0.01, P = 0.94). Moreover, no significant difference in task performance has been found between the subjects who consistently used the same hand pose combinations and those who changed hand poses across trials (average finger movement time: F_1,48= 1.49, P = 0.23, average failure rate: F_1,48 = 0.92, P = 0.34). Thus, task performance does not seem to be affected by the change of hand poses.

Structural similarities and differences of hand pose matrices.

In both the free-base and the fixed-base conditions, similar choices of execution and localization hand pose combinations were observed. In the free-base condition, 6 out of 10 subjects maintained the same hand pose combinations throughout all trials (Fig. 9B), whereas in the fixed-base condition, only three subjects used the same hand pose combinations (Fig. 11B). The majority of subjects showed diversity in choices and used at least two different hand pose combinations. This observation was quantified by the structural similarity index s, which assessed similarities across distributions of hand pose combinations during localization and execution in the two experimental conditions. We found the highest similarity score between localization and execution steps within the same experimental condition: $s(H_{\mathrm{f}\mathrm{r}\mathrm{e}\mathrm{e}}^{\mathrm{l}\mathrm{o}\mathrm{c}},H_{\mathrm{f}\mathrm{r}\mathrm{e}\mathrm{e}}^{\mathrm{e}\mathrm{x}\mathrm{e}})$=0.62 for the free-base condition and $s(H_{\mathrm{f}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{d}}^{\mathrm{l}\mathrm{o}\mathrm{c}},H_{\mathrm{f}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{d}}^{\mathrm{e}\mathrm{x}\mathrm{e}})$= 0.18 for the fixed-base condition. The low similarity scores $s(H_{\mathrm{f}\mathrm{r}\mathrm{e}\mathrm{e}}^{\mathrm{l}\mathrm{o}\mathrm{c}},H_{\mathrm{f}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{d}}^{\mathrm{l}\mathrm{o}\mathrm{c}})$ = 0.09 and $s(H_{\mathrm{f}\mathrm{r}\mathrm{e}\mathrm{e}}^{\mathrm{e}\mathrm{x}\mathrm{e}},H_{\mathrm{f}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{d}}^{\mathrm{e}\mathrm{x}\mathrm{e}})$= 0.07 indicated that hand pose combinations used in both experimental steps in the free-base condition were distinctly different from those in the fixed-base condition.

Virtual fingers assignment.

Specific patterns in the hand pose combinations’ distribution in hand pose matrices could be observed when paying attention to the functionality of hand poses. This functionality could be assessed by analyzing the virtual finger (VF) assignment. In Fig. 13, hand pose combinations during execution were regrouped according to their VF assignment. As stated previously, in the free-base condition, subjects displayed consistently similar hand pose combinations with similar VF assignment. Figure 13A illustrates this by showing all hand pose combinations located in the intersection region. The right hand formed three VFs, all of which were dedicated to the control of both task demands (highlighted in yellow). The other hand was in charge of compensating for eventual perturbations, but did not dedicate VFs to controlling explicit task demands (highlighted in gray).

Figure 13.

Assignment of virtual fingers in the observed execution hand pose combinations and their controlled task demands in the free-base condition (A) and the fixed-base condition (B). Hand pose combinations in the small figures correspond to the categories in Fig. 5. The shade of the color bar is proportional to the values. The hand pose combinations that were only used to control the perturbation are highlighted in gray color; hand pose combinations that are assigned 1, 2, and 3 virtual fingers are highlighted in green, red, and yellow color, respectively. The number of trials in which such hand pose combinations were used are denoted as percentage. VF, virtual finger.

In the fixed-base condition, the pattern of VF assignment was different. VFs in charge of controlling task demands were distributed across the two hands. In 60% of the experimental trials, one of the two hands used one VF to generate the required vertical force (highlighted in green), whereas the other hand used two VFs to control rotating torques (highlighted in red, Fig. 13B). These hand pose combinations correspond to the entries $H_{\mathrm{f}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{d}}^{\mathrm{e}\mathrm{x}\mathrm{e}}$(4–5, 1) and $H_{\mathrm{f}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{d}}^{\mathrm{e}\mathrm{x}\mathrm{e}}$(1, 3–4). In 20% of trials, corresponding to $H_{\mathrm{f}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{d}}^{\mathrm{e}\mathrm{x}\mathrm{e}}$(3–4, 8), the thumb and middle finger from the right hand only played a secondary role to assist the torque control by the lead hand. If only the degrees of freedom of the task are considered and the assisting fingers are ignored, then these results show that subjects divided force and torque across the hands in 80% of the trials. Subjects used one hand to control one independent task demand. This contrasts to the free-base condition where one hand controls both task demands (intersection of highlighted regions, Fig. 13A).

DISCUSSION

To increase our understanding of how different hand poses are selected and combined in a bimanual high-precision manipulation task, this study examined how humans maneuvered a jeweler’s screwdriver to dismount a watch screw. Asking subjects to perform the task under two task conditions with different numbers of degrees of freedom to control for allowed to determine the effects of task’s demands on their hand pose selection strategies. We hypothesized that tasks with more degrees of freedom for hand allocation would lead to more variability in hand pose combinations. In contrast, tasks with fewer opportunities for variations in hand poses will enhance the influence of hand dominance in role distribution.

As expected, subjects used distinct hand pose combinations to satisfy the control of task demands under both conditions. The fixed-base condition was less constrained and subjects displayed a larger variety of hand pose combinations. In contrast, in the free-base condition, subjects needed to maintain the watch face in a stationary position on the table, which required the control of additional degrees of freedom. These constraints limited hand pose variations and only a few hand pose combinations could satisfy the control of task demands and successfully complete the task.

We also observed a strong preference for bimanual over unimanual operation when completing the task. In the free-base condition, the left hand provided support by controlling the residual degrees of freedom, while the right hand performed the major manipulation motion. However, this role distribution was modulated when the task’s degrees of freedom decreased in the fixed-base condition, allowing for more variations of hand pose combinations. The assisting left hand started to contribute more actively to generate the movement, either by coordinating its finger movements or by balancing the forces with those of the right hand that was in charge of generating motion. At times, roles across the two hands were switched as subjects explored different hand poses.

To display the diversity of hand pose combinations and ease visual assessment of the role distribution across the two hands, we created a novel visualization approach, the hand pose matrix. Sparseness of the matrix entries denoted low diversity in hand pose combinations and vice versa. The prevalence of handedness was expressed in the concentration of entries in the upper and lower diagonals in the matrix. To account for hand poses particular to watchmaking, we expanded the extant taxonomies of hand poses. To relate our taxonomy to the task-specified control demands, hand poses were categorized according to their virtual finger assignment in relation to force/torque control abilities. This matrix representation helped to summarize and quantify the unbalanced role distribution across the two hands. This matrix representation is not restricted to this study and can be applied to analyze general bimanual tasks.

Trade-off between Speed and Accuracy

An analysis of task performance in terms of movement time and failure rate revealed that a decrease in task completion time was associated with an increase in failure rate. This observation is consistent with the widely observed speed-accuracy trade-off, i.e., improvements in accurate performance are achieved at the expense of speed, or vice versa (57–61). This basic finding validated that these realistic and complex data conform to generally accepted performance features.

Task’s Degrees of Freedom and Their Effect on Hand Pose Selection

Our study contrasted two task conditions to quantify the effect of the task’s degrees of freedom on the selection of hand poses. The free-base condition increased the number of degrees of freedom to be controlled in the task, due to the necessity of maintaining the watch face in a stationary position. In this condition, less variability in the hand pose combinations was observed in both the localization and the execution task steps.

Experimental condition: free-base condition.

Without exceptions, subjects in the free-base condition adopted the same pose combinations during both task steps with their left hand to stabilize the watch face on the table (Figs. 7A and 9A). Given the extra degrees of freedom imposed by the watch face, pose 2 was the only feasible hand pose when the supporting (left) hand controlled these degrees of freedom to maintain the watch face’s stability. Using hand pose 2 to control the degrees of freedom from the watch face was, however, not the determining factor for a successful task completion because it did not generate the force and torque required to rotate the screwdriver. Force and torque were then to be generated by the dominant (right) hand. In principle, the right hand could use any hand poses (except poses 1 and 2) to generate the task-demanded force and torque. However, the subjects solely used hand poses 8 and 9. This may be attributed to the fact that hand poses 3–6 led to inconsistent force generation, since these four hand poses did not have a VF specified for the control of force. When using these poses, the only way to generate the necessary force was pressing the screwdriver downward while rotating them. Hand poses 7–10 used an extra virtual finger for force control, which corresponded to either the index finger (for poses 8–10) or the palm (for pose 7). These poses delivered constant pressure without affecting rotational movements. Interestingly, subjects did not use hand pose 7 at any stage, despite its feasibility for controlling both task demands. This could be attributed to fatigue caused by lifting the forearm or bending the wrist that was required when using this hand pose. Holding the object in place while inserting a screw is part of a wide range of tasks, including repairing a cell phone, a watch, or glasses. This use of hand pose in the free-base condition may reflect the subjects’ habits acquired through life experience (35).

Experimental condition: fixed-base condition.

In the fixed-base condition, the watch face was mounted on the table. This provided more options for both hands and thus led to a wider selection of hand pose combinations, as quantified by larger entropy values $h_{\mathrm{f}\mathrm{r}\mathrm{e}\mathrm{e}}^{\mathrm{l}\mathrm{o}\mathrm{c}}$and $h_{\mathrm{f}\mathrm{r}\mathrm{e}\mathrm{e}}^{\mathrm{e}\mathrm{x}\mathrm{e}}$. Primary differences were in the assignment of function: hand poses separated the control of the force and torque across the two hands. In ∼60% trials, one hand (in pose 1) provided downward pressure on the screwdriver, whereas the other hand rotated the screwdriver.

The observed combinations of hand poses in the free-base condition were rarely seen in the fixed-base condition. Specifically, hand pose 2 was used in the free-base condition to restrain the watch face and was only used in 3 out of 50 trials by 1 particular subject (only in combination with pose 7 from the right hand). In addition, unimanual manipulation manner was used in 14% trials with the right hand having the similar functionality as being used in the free-base condition.

Synergies, decoupling of control variables, and hand pose selection.

The concept of synergy has been widely used to explain human hand pose, preshape, and finger coordination in humans (50) and to control artificial hands in robots (62). It assumes that the human brain couples hand joints or muscles and controls them in a lower-dimensional space through synergies. Task-specific synergies can be organized at multiple levels, from neural constructs to muscular coordination patterns. There is a wealth of evidence that some synergies develop during childhood, such as the simultaneous control of all fingers for a power grasp and the coordinated control of thumb and index for a pinch grasp. Such simultaneous control of multiple degrees of freedom, although beneficial to speed up daily object manipulation, limits the independent control of fingers that is necessary for fine manipulation, as found in crafts and playing musical instruments.

In addition to synergies, there is evidence that fingers are coupled biomechanically. Tendons cross multiple finger joints and finger range of motion is limited by soft tissues. Individual movements of each finger are impeded by the fact that they share the same group of muscles or are connected by tendons (63, 64). In addition to the anatomical linkages (65), the specific innervation of adjacent fingers leads to synchronous flexion of adjacent finger joints (66). Such biomechanical coupling affects also the ability to generate force independently with each finger and force applied by one finger may inadvertently lead other fingers to also passively exert force. This is known as the force-enslaving effect (see Ref. 67 for a review). In our study, such biomechanical coupling affects performance. Indeed, control of the thumb, the middle, and the ring fingers to generate the rotation of the screwdriver inevitably affects the control of the index finger and makes it more challenging for this finger to maintain a steady and stable downward pressure on the screwdriver to stabilize it. Such interference among fingers in the same hand can be largely reduced, if all fingers work in coordination to produce the same movement, i.e., mapping all fingers to the same VF. This speaks in favor of control solutions that reduce the number of VFs required for the task, as observed in our data. Therefore, subjects’ choice of hand poses in the fixed-base condition is likely the result of the brain trying to reduce undesired variability caused by biomechanical couplings.

Individualized finger control is demanding and requires practice. To achieve individual control of the fingers requires decoupling this neural organization and activating additional muscles to minimize unintended movements of other fingers (67). Despite these hurdles, dexterous finger movements can be attained through practice as evident in the extraordinary skills of pianists (68, 69). Intensive practice of these skills can lead to change in the motor cortex’s representation of finger digits (70). Although producing certain finger coordination patterns becomes easier with training, generating such novel coordination increases intrinsic computational burdens, especially when a large number of degrees of freedom need to be controlled simultaneously, as it is the case in bimanual fine-manipulation tasks. One may hence prefer to not train novel finger coordination patterns or synergies. Instead, one usually would seek alternative approaches, such as decoupling the control variables required for the task into simple and independent components. In this study, the task afforded the decoupling of force and torque production. Such decoupling may simplify control and allow to use hand postures from an existing synergy repertoire. Control of each hand is then made easier by the fact that each of these primary hand poses controls one independent function. In our task, however, such decoupling requires the control of both hands. Coordinating both hands to perform the task is not necessarily easier than learning new coordination patterns, especially when this coordination is unusual.

In the free-base condition, subjects used extensive hand pose combinations with the motion-control hand having three VFs. Controlling these three VFs requires subjects to adopt task-specific postural synergies, some of which may be novel or less trained depending on the subjects’ life experience. In contrast, in the fixed-base condition, most hand poses required control of one or at most two VFs. Moreover, most of these hand poses are typical of those adopted in everyday tasks (41). Thus, commonly used synergies may be sufficient for controlling such hand poses. This study did not evaluate cognitive load required to perform the task in either conditions. We can however expect that subjects may have found the free-base condition more challenging to control.

Hand Pose Selection Strategies across Localization and Execution

In an effort to determine whether hand pose selection was influenced by sequential variations in task demands, we also compared hand pose combinations across the two steps of our task: localization and execution. Recall that localization required to control precise positioning of the watch face and the tool, whereas execution denoted the segment where the hands controlled both force and torque on the tool, in addition to movement.

In the free-base condition, we did not observe major changes in hand poses from the localization to execution due to the constraints imposed. Hand dominance seemed to be the primary factor to assign roles to the left and right hands in both steps. Moreover, although most subjects used the same left hand pose (pose 2) in both steps, they modified the right hand pose when transitioning from localization to execution (from pose 9 to pose 8, and from pose 10 to pose 9) by decreasing the number of active fingers. Having more fingers in contact with the screwdriver better guaranteed stability but imposed more constraints on its movement. This was valuable during localizing the tool but detrimental during execution, as the tool had to slide along more fingers. In addition, coordinating multiple fingers to achieve the fast rhythmic rotating movement proved more difficult with several fingers.

In the fixed-base condition, subjects used the dominant right hand to localize the tool in almost all (98%) trials. Most subjects adopted similar hand pose combinations from localization to execution with only slight adjustments of finger placements. For example, one subject (S7) removed one active finger from each hand after transiting to the execution step in all trials. Another subject (S1) did not use the index finger during localization, and then placed the index finger as one independent VF to control force during execution. Although most modifications reduced the number of active fingers, it did not change the roles and functions of the two hands, as the VFs and their specific functions remained the same. This was likely due to the fact that both task steps required precision. Only 2 out of 10 subjects, S6 and S10, adopted consistently the identical hand poses across the 2 experimental steps without any adjustment.

Our analysis revealed that subjects tended to adopt the same type of hand pose for both steps. Although we observed differences in hand poses across localization and execution steps, these were minor changes overall. This is in line with the observation that, despite the seemingly infinitely many controlled variables, humans appear to consistently choose the same types of hand poses for similar tasks and to do so effortlessly (6). In our study, preserving the same hand pose throughout the task appeared more time efficient and less cumbersome, as it obviated repositioning the screwdriver or adjusting finger placement on the tool. This choice was likely dictated by an economy of efforts and a desire to increase comfort (71). This observation is consistent with evidence showing that hand poses are primed differently depending on how to maneuver a tool to accomplish a skillful task (15). Our study demonstrated that this mental preshaping, shown previously in unimanual tasks, also applied to bimanual tasks.

Hand Dominance Determines Role Distribution

As expected, hand dominance was a factor to determine functions across the two hands (25). The dominant hand adopted the role that required finer control of forces and torques. Dominance was particularly visible when the task degrees of freedom were high as in the free-base condition, but also during localization in both conditions. Interestingly though, in the fixed-base condition, hand dominance seemed to play a lesser role during the execution phase, as in only 60% of the trials, the right hand rotated the screwdriver, whereas the left hand did so in 40% of the trials. In 20 trials, subjects even swapped hands, using the left instead of the right hand for guiding the screwdriver. In 10 trials, the two hands even alternated when rotating the screwdriver. Six subjects used their left hand instead of their dominant right hand to control motion in tasks. Additional analyses did not reveal any significant difference in the handedness index between the subjects who swapped hands and others (F_1,8= 1.05, P = 0.3364). Moreover, subjects’ failure rate and movement time also do not relate to their handedness indices (average failure rate: F_1,8= 2.56, P = 0.1483, average movement time: F_1,8 = 2.47, P = 0.1547).

Manipulating the screwdriver and trying to avoid cam-outs require high accuracy of control. As suggested by the dynamic dominance hypothesis (29), the dominant limb is specialized for controlling the task dynamics, whereas the nondominant limb achieves higher positional accuracy (30, 32–34). Therefore, swapping roles between hands could favor the nondominant (left, positioning the watch face) hand’s control for improving the accuracy by attenuating the dominant (right, controlling the motion of screwdriver) hand’s control strength. This swap may also be a strategy to promote task effectiveness by avoiding reorienting or regrasping the screwdriver, as suggested by Theorin and Johansson (38). According to our observation, in these switched trials, subjects shaped their left hand in poses 3, 4, or 5 during the localization step. If the left hand played the lead role during localization in some of these trials, the subject would have used the left hand as the prime actor in the following execution step without reassigning the roles of both hands.

Why Bimanual Control When Unimanual Control Suffices?

Our results showed that subjects preferred to use both hands, even when a single hand would have been sufficient to complete the task. In the fixed-base condition, the task could have been performed with one hand. However, unimanual manipulation manner was observed in only 14% trials [Fig. 11A, $H_{\mathrm{f}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{d}}^{\mathrm{e}\mathrm{x}\mathrm{e}}$(0, 7–9)], and in most (86%) trials, subjects still performed the task bimanually.

Such preference for using two hands can be observed in numerous daily life activities. For example, a bottle cap can be unscrewed with one hand. However, doing this operation unimanually is more complex from a control viewpoint. First, it requires control of the palm, little finger, ring finger, and sometimes the middle finger to enwrap the bottle. Then, as the palm and smaller fingers stabilize the bottle’s neck, the index and thumb must rotate in synchrony to unscrew the cap. Because of the biomechanical coupling and neural cross talk, to generate thumb and index movements may lead to a decrease in the force applied by the other fingers and to destabilize the object. The same task, however, can be achieved with more accuracy when using both hands.

Our data indicates that the same principle is at play. Unimanual control of the screwing task required independent control of the index finger from the other fingers, as well as to control up to three VFs to generate the required force and torque simultaneously. This forced to break the traditional thumb-finger coupling, which may have contributed to the difficulty of performing the task with a single hand (41, 72). Instead, when using both hands, torque control can be decoupled from force control, and one needs only to use the common thumb-index coupling to generate the desired motion on the tool.

Efforts required to decouple the control of task demands across the two hands paid off. Indeed, when examining task performance and efficiency, the results showed that bimanual task execution was faster than unimanual task execution. Subjects moved their fingers significantly faster, while maintaining similar average failure rates. Selection of unimanual or bimanual strategy may be dictated by a minimization of cost (38). Subjects may have opted for bimanual over unimanual control in an attempt to maximize success and reduce failure rate, in place of optimizing for economy of efforts.

Although decoupling and distributing the task across the two hands may seem to simplify the overall control, one should not forget that this now requires to coordinate the two hands. In our study, likely little effort was required to coordinate the two hands, since one of the two hands was static, fixating the watch or holding the screwdriver. The brain could hence rely on existing bimanual strategies for spatiotemporal coordination inherent to bimanual tasks (73). In our study, bimanual coordination was reduced to spatial coordination, as both hands had to align the tool with the required position and orientation relative to the watch. Control of the screwdriver’s orientation and position required high precision to avoid cam-outs. This was made particularly challenging as subjects had to rely more on proprioception than vision, given the size of the screw and the fact that the screwdriver could sometimes block the view on the screw. Using both hands was likely advantageous over using a single hand, as contact between the hands could mitigate sensorimotor noise and improve spatial localization (74, 75).

Subjects’ preference for bimanual over unimanual control was hence likely an effort to both reduce variability caused by biomechanical couplings and to alleviate intrinsic sensorimotor processing burdens due to the large number of degrees of freedom in both hands that needed to be controlled. Since both hands were tasked to work on the same spatial localization task, efforts may also have been reduced (38).

Conclusions

This study presented evidence that a task’s number of degrees of freedom play an important role in the selection and shaping of bimanual hand poses. The results suggested a strong preference for bimanual over unimanual operation that would require a decoupling of individual fingers’ control. The choice of hand poses is informed by handedness; the dominant hand performs the major movement components, and the nondominant hand tends to assist and control residual degrees of freedom. However, as tasks become less constrained (task’s degrees of freedom reduced), the nondominant hand contributes more actively and shares forces with the dominant hand. At times, the roles of the two hands were switched as subjects explored different hand poses. Subjects exhibited a diversity of hand pose combinations when the task became less constrained. We speculate that this search for different hand poses may be an effort to improve performance, possibly seeking postures that minimize the influence of sensorimotor noise on precision of control (76). Although our study focused on a bimanual task, our results also offer insights into unimanual performance. Interestingly, subjects rarely exploit the full potential of the hands’ and fingers’ dexterity.

To facilitate visual assessment of the diversity and role distribution of hand poses, we proposed a hand pose matrix as a tool to summarize experimental results. This visualization is not specific to this study and may be useful for other bimanual tasks. The taxonomy also included analysis of virtual fingers in relation to control of motion and force/torque distribution demanded by the task. It is worth noting that the hand pose combinations used in this study are task specified and were created for distinct task conditions. It does not include other hand poses found in extant hand taxonomies that were not relevant in the present context. The hand pose matrix does not provide information regarding the change of hand poses across trials. It would be interesting to extend this matrix to include a temporal tracking of hand poses to provide time-varying task information.

Finally, our analysis did not relate the choice of hand pose to the force and torque required by the task, mainly because our tasks required only small forces that are well within the strength of the human hand. Moreover, it is also suggested that the control of hand postures and the regulation of contact forces are independent (50). However, for tasks that require larger forces, such as opening the cap of a bottle, the selection of hand poses and role distribution is likely modulated by the precision and strength that each hand can contribute. Analyzing applied forces for manipulating the same object with different hand poses may provide more insights on the effects of force demands.

GRANTS

This project was supported through a European Research Council Advanced Grant, project ID: 741945, funded by the European Commission, awarded to A. Billard. D. Sternad was supported by the National Institutes of Health, R01-HD087089 and R01-CRCNS-NS120579, and by the National Science Foundation, M3X-1825942 and NRI-1637854.

DISCLOSURES

No conflicts of interest, financial or otherwise, are declared by the authors.

AUTHOR CONTRIBUTIONS

A.B. conceived and designed research; K.Y. performed experiments; K.Y. analyzed data; K.Y., D.S., and A.B. interpreted results of experiments; K.Y. prepared figures; K.Y. drafted manuscript; K.Y., D.S., and A.B. edited and revised manuscript; K.Y., D.S., and A.B. approved final version of manuscript.

ENDNOTE

At the request of the authors, readers are herein alerted to the fact that additional materials related to this manuscript may be found at https://www.epfl.ch/labs/lasa/sahr/databasecode/. These materials are not a part of this manuscript and have not undergone peer review. American Physiological Society (APS) and the journal editors take no responsibility for these materials, for the website address, or for any links to or from it.