Abstract
Tablet computers’ hardware and software designs may affect upper extremity muscle activity and postures. This study investigated the hypothesis that forearm muscle activity as well as wrist and thumb postures differ during simple gestures across different tablet form factors and touchscreen locations. Sixteen adult (8 female, 8 male) participants completed 320 tablet gestures across four swipe locations, with various tablet sizes (8” and 10”), tablet orientations (portrait and landscape), swipe orientations (vertical and horizontal), and swipe directions (medial and radial). Three-dimensional motion analysis and surface electromyography measured wrist and thumb postures and forearm muscle activity, respectively. Postures and muscle activity varied significantly across the four swipe locations (p< 0.0001). Overall, swipe location closest to the palm allowed users to swipe with a more neutral thumb and wrist posture and required less forearm muscle activity. Greater thumb extension and abduction along with greater wrist extension and ulnar deviation was required to reach the target as the target moved farther from the palm. Extensor Carpi Radialis, Extensor Carpi Ulnaris, Flexor Carpi Ulnaris, Extensor Policis Brevis, and Abductor Pollicis Longus muscle activity also increased significantly with greater thumb reach (p<0001). Larger tablet size induced greater Extensor Carpi Radialis, Extensor Carpi Ulnaris, Flexor Carpi Ulnaris, Flexor Carpi Radialis, and Abductor Pollicis Longus muscle activity (p<0.0001). The study results demonstrate the importance of swipe locations and suggest that the tablet interface design can be improved to induce more neutral thumb and wrist posture along with lower forearm muscle load.
Keywords: User Interface Design, Electromyography, Motion Capture, Mobile Computing
Introduction
Technology users are moving away from stationary computer workstations and migrating to portable units such as the tablet computer because of their mobility and functional versatility. In fact, the term “phablet” has emerged for the class of mobile devices designed to combine or bridge the form of a smartphone and a tablet (Hill, 2013). While these mobile devices are designed to be multi-functional with an often intuitive software interface, their designs may challenge users’ biomechanical capabilities and may be associated with musculoskeletal disorders (MSD) caused by overuse. Several studies have explored how tablet design and configuration affect biomechanical factors as well as user experience and performance related issues with these devices.
There is evidence that certain display and hand holding configurations of tablet use is associated with neck and head flexion, as well as wrist extension (Trudeau, Young, Jindrich, & Dennerlein, 2012; Young J., Trudeau, Odell, Marinelli, & Dennerlein, 2012; Young J., Trudeau, Odell, Marinelli, & Dennerlein, 2013). Grip and input technique have been shown to affect forearm muscle loading and performance (Gustafsson, Johnson, Lindegard, & Hagberg, 2011). Pereira et al. found that during one- handed smaller to medium-size tablet use, using a ledge or handle on the back was associated with greater overall usability compared to the no-handle condition (2013).
Both tablet orientation (portrait/landscape) and its touch keyboard layout can significantly affect users’ thumb posture, perceived-comfort, and motor performance while performing a tapping task during two-handed use of tablet computers (Trudeau, Catalano, Jindrich, & Dennerlein, 2013). Trudeau (2016) found that two-handed grips afforded better performance and greater wrist and proximal thumb joint extension when compared to one-handed grips.
Swiping and tapping are similar gestures that both require the user to identify and reach for a target, but swiping requires prolonged contact with the touchscreen surface to reach a secondary target (Villamor, Willis, & Wroblewski, 2010). The user interface (UI) affects biomechanics and motor performance for both gestures. Studies have explored how upper extremity biomechanics and performance are affected by factors such as touch target size and location during tapping tasks (Park & Han, 2010; Ko, Hwang, & Liang, 2016; Jeong & Liu, 2017). In addition, Jeong & Liu (2017) found horizontal swipes using the index finger had better self-reported performance and lower physical demands than vertical swipes.
Users also use their thumbs to complete swipe gestures (Billinghurst & Vu, 2015). While a few studies have explored thumb functional reach on touch screen devices (Bergstrom-Lehtovirta & Oulasvirta, 2014; Odell & Chandrasekaran, 2012), to our knowledge, there has been no investigation of how different tablet form factors, UI designs, and tasks affect thumb biomechanics and motor performance for swipe gestures.
In the current study, we sought to determine the effect of tablet form factor (size, orientation) and swiping gesture design (location, orientation, direction) on thumb swiping motor performance, thumb posture, forearm muscle activity, and user perception across configurations for a two-handed grip on a tablet device. Similar to typing performance, we expect that swiping performance, self-reported discomfort, and forearm muscle activity would differ across the gesture designs and form factors due to different thumb and wrist postures required to perform swiping tasks. Specifically, we hypothesize that right thumb swiping close to the palm in the bottom right location of the tablet’s screen would require less reach than further from the palm in the middle of the screen and that we can measure this as 1) lower forearm muscle activity; 2) lower self-reported discomfort; 3) better performance compared to the top left location.
Methods
Study Population
Sixteen healthy right-handed participants (8 males and 8 females, aged from 21 to 40 years old) with no history of upper-extremity MSDs were recruited for the study (Table 1). The Harvard T.H. Chan School of Public Health and Northeastern University institutional review boards approved all protocols and informed written consent forms. For the testing protocols, participants sat on a height adjustable task chair without arm supports. The height of the chair was adjusted while their feet were flat on the floor and their thighs were horizontal with their knees at a ninety-degree angle. All nearby light sources were indirect, and there was no glare on the tablet’s screen. Participants were instructed to hold the tablet with two hands and interact with the tablet using only their right thumbs without dropping the tablet. Participants sat upright and were allowed to use their laps (thighs) to help support their hands/wrists/distal forearms while using the device. The entire experiment including set-up took approximately two hours.
Table 1.
Anthropometric measures of means (standard deviations) across all participants
| Males (N=8) | Females (N=8) | All | |
|---|---|---|---|
| Age (yrs) | 25 (4) | 24 (3) | 24.5 (3) |
| Height (cm) | 180 (8) | 167 (7) | 173.4 (10) |
| Weight (kg) | 74 (20) | 61 (6) | 68 (14) |
| Hand Length (cm) | 20 (0.8) | 18 (1) | 19 (1) |
| Hand breadth (cm) | 8.8 (0.6) | 7.4 (0.4) | 8.1 (1) |
| Thumb length (cm) | 10 (1) | 9.7 (1) | 10 (1) |
Tablet Instrumentation and Experimental Tasks
Participants performed 16 thumb swiping gestures with the thumb of their right hand, repeating each gesture 5 times in four different tablet configurations for a total of 320 swipes. The swipe gestures differed in swipe direction (outward vs. inward), swipe orientation (horizontal vs. vertical), swipe location (4 swipe zones), and swipe length (short vs. long). Tablet configurations differed in tablet size (small vs. large) and tablet orientation (portrait vs. landscape) (Table 2). As defined in Trudeau (2012), “outward” movements of the thumb were defined as consisting primarily in carpometacarpal (CMC) joint flexion or abduction movements with extension of the interphalangeal (IP) and metacarpal (MCP) joints and include the following directions: South(S) → North (N) & East(E) → West(W). “Inward” movements of the thumb were defined as consisting primarily in CMC extension or adduction movements with flexion of the IP and MCP joints and include the following directions: N → S, NW → SE, W → E, and SW → NE.
Table 2 –
Different swipe design characteristics considered in the tablet study. The study protocol consisted of four sets of the presented swiping actions randomized across two different tablet sizes (8” and 10”) and orientations (portrait and landscape).
| # | Orientation | Length | Direction | Starting location |
|---|---|---|---|---|
| 1 | Horizontal | Long | Inward | 1 |
| 2 | Horizontal | Long | Outward | 2 |
| 3 | Horizontal | Long | Inward | 3 |
| 4 | Horizontal | Long | Outward | 4 |
| 5 | Vertical | Long | Inward | 1 |
| 6 | Vertical | Long | Outward | 3 |
| 7 | Vertical | Long | Inward | 2 |
| 8 | Vertical | Long | Outward | 4 |
| 9 | Horizontal | Short | Inward | 1 |
| 10 | Horizontal | Short | Outward | 1 |
| 11 | Horizontal | Short | Inward | 2 |
| 12 | Horizontal | Short | Outward | 2 |
| 13 | Horizontal | Short | Inward | 3 |
| 14 | Horizontal | Short | Outward | 3 |
| 15 | Horizontal | Short | Inward | 4 |
| 16 | Horizontal | Short | Outward | 4 |
The swipe gesture required the user to move a cursor along and within a lane created by two lines a specified distance apart (10 mm) for a specified distance (short 20 mm or long 60 mm). To complete the swipe gesture, participants had to touch the screen activating a target bar (10mm × 2mm) in the center edge of one of the four zones (Figure 1) and then steer the bar between two lines while keeping the thumb between the two lines (Accot & Zhai, 1997; Dennerlein, Martin, & Hasser, 2000). The gesture was completed when the thumb reached and passed the end of these lines without movement going outside the lane formed by these two lines. Each gesture was performed 5 times in the same direction as shown on the screen. The subjects were instructed to swipe in the direction each time and to naturally bring their thumb back to their starting position without swiping backwards on the screen. No time limit was set for each participant as they were only instructed to complete each trial as quickly and accurately as possible. Each 5 trial task typically took around 1 minute. A custom native application was created for an Android program to collect completion time data and provide visual guidance for users.
Figure 1a and 1b.
Portrait and landscape swipe locations on the screen. Thumb swipe lanes were indicated by a target bar (10mm × 2mm) starting in the center edge of each of the four zones. Long vertical and horizontal swipes (60 mm) crossed zones and short horizontal swipes (20 mm) remained in one zone.
The two tablet computers selected in the study were a Samsung Galaxy III with a 10” display (Samsung Electronics Co., Ltd., South Korea) and a Samsung Galaxy Note III with an 8” display (Samsung Electronics Co., Ltd., South Korea). Device, device orientation, and each thumb swipe task were presented to the participant in a balanced randomized fashion (Figure 1).
Dependent Variables: Posture
Participants’ hand and upper limb postures were calculated from data recorded using an optical three-dimensional motion analysis system (Optotrak Certus, Northern Digital, Ontario, Canada). Infrared light-emitting diodes (IRLEDs) were mounted on the tip, interphalangeal joint (IP), metacarpal (MCP) and carpometacarpal (CMC) joint of the participant’s right thumb. A rigid body cluster consisting of three IRLEDs was attached to a metal structure and attached to the back (dorsal) side of the right hand over the 3rd metacarpal bone between the wrist and MCP joint. A second rigid body was attached to the dorsal side of the right forearm approximately two inches proximal to the right wrist (Trudeau, Catalano, Jindrich, & Dennerlein, 2013). Locations of bony landmarks (lateral and medial epicondyles, lateral and medial styloids, and metacarpophalangeal joints for digits II-IV of the right hand) were palpated, digitized, and tracked corresponding to the hand rigid body. Location data for each IRLED and digitized point were subsequently filtered through a low-pass, fourth-order Butterworth filter with a 10 Hz cutoff frequency to define local coordinate systems of the thumb, hand, and forearm. The IRLED placement used in this study builds on previous methods for measuring thumb-tablet interactions (Asundi, Odell, Luce, & Dennerlein, 2012; Trudeau, Catalano, Jindrich, & Dennerlein, 2013; Winter, 2005).
The wrist and thumb joint angles were calculated using the Euler angles of the rotation matrices describing the orientation of the joint’s distal segment relative to the proximal segment (Winter, 2005). We presented joint angles relative to a reference posture in which forearm and hand were aligned along its longitudinal axis, and the thumb was extended and straightened to the lateral side of the index finger. This reference posture was chosen because it is unambiguous and repeatable. The posture results were based on the assumption that the wrist has two degrees of freedom (radial/ulnar deviation, flexion/extension), the thumb CMC joint has three degrees of freedom (abduction/adduction, flexion/ extension, pronation/supination), the thumb MCP joint has two degrees of freedom (abduction/adduction, flexion/extension), and the thumb IP joint has a single degree of freedom (flexion/extension). Median joint angles and joint ranges of motion (90th minus 10th percentile) were calculated as metrics to describe hand and thumb posture for each trial (Dennerlein & Johnson, 2006; Asundi, Odell, Luce, & Dennerlein, 2012; Trudeau, Catalano, Jindrich, & Dennerlein, 2013).
Dependent Variables: Muscle Activity
Surface electromyography (EMG) electrodes (DE-2.1 Single Differential Electrode; Delsys, Boston, Massachusetts, USA) measured 6 muscle groups of the right forearm including Extensor Carpi Radialis (ECR), Extensor Carpi Ulnaris (ECU), Extensor Pollicis Brevis (EPB), Abductor Pollicis Longus (APL), Flexor Carpi Radialis (FCR), and Flexor Carpi Ulnaris (FCU). The electrodes were placed in standard locations based on previous experience and as defined by Perotto (Perotto, 1994; Lin, Young, & Dennerlein, 2015). Electrode placement on the muscles was achieved through palpation and validated through EMG signal response to corresponding muscle contraction exercises. The EMG sampling rate was 1000 Hz. Upon amplification, EMG signals were rectified and smoothed using a 3 Hz low pass filter. Three isometric maximum voluntary contractions (MVC) were collected for each muscle to normalize EMG signals within participants. Participants were coached to gradually ramp up to reach their maximal exertions for 3 seconds while the experimenter resisted using up to their entire bodyweight. The highest value of the 3 exertions was designated as the 100% MVC. A minimum break of at least two minutes was taken between MVC of each muscle group. Based on these references, normalization of EMG was calculated as percent MVC of each muscle. Median (50th percentile) muscle activity levels in percent MVC were used to compare across participants.
Dependent Variables: Performance
The custom native Android program collected completion time data for the five swipes in each of the configurations.
Dependent Variables: Self-reported discomfort
All participants responded to eight survey questions about overall hand and wrist discomfort after completing each condition with each tablet. The responses were marked on a 10-cm visual analogue scale (VAS) with 0 cm being the lowest level of perceived-discomfort and 10 cm being the highest.
Data and Statistical Analysis
A repeated measures ANOVA (RMANOVA) evaluated the effect of each independent variable. A single ANOVA model was run for each of the 16 dependent variables, including posture (8, in angles), muscle activity (6, in percentage MVC), performance (1, in milliseconds) and user perception (1,VAS scale from 0 to 10).. For each ANOVA model, we included participant as a random effect and all five independent variables (tablet size (1df), tablet orientation (1df), swipe direction (1 df), swipe location (3 df), swipe orientation (1df)) as fixed effects, as well as all possible two-way interaction terms. None of the three-way and higher interaction terms were significant. The significance criterion (alpha value) was set at 0.05. When a significant effect was found, either a post-hoc analysis with Tukey’s honest significance test for variables with more than two levels or a student’s t-test was conducted for variables with two levels. Statistical analysis was performed using JMP Pro 11 (SAS) linear mixed model software.
Results
Posture
Thumb postures differed significantly across the four swipe locations (Table 3a). Specifically, the top left swipe location was associated with less IP flexion (6°), MCP flexion (12°), and CMC pronation (0°), but greater MCP and CMC abduction (16° and 7°, respectively). The bottom left location was associated with the greatest CMC abduction (10°), compared to other locations. The bottom right location was associated with greater IP flexion (40°), MCP flexion (20°), and CMC pronation (3°), but smaller MCP and CMC abduction (10° and 0°, respectively) than the top left location. Additionally, thumb swipes in the top left and the bottom left swipe zones required greater ranges of thumb movement compared to the top right and the bottom right swipe zones (Table 3b). Horizontal swipes were associated with greater MCP abduction (15°) and flexion (18°) and CMC abduction (4°), compared to vertical swipes (Table 3a). There were no significant differences in thumb posture across different hardware configurations.
Table 3a.
Thumb and wrist posture: Across participant marginal means (and standard errors) for joint angles (°). Joint angles were expressed relative to a reference posture where the forearm, hand, and fingers were aligned along the longitudinal axis, and the thumb was extended and straightened to the lateral side of the index finger.
| Tablet Size | Tablet Orientation | Swipe Zone2 | Swipe Orientation | Swipe Direction | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| P-value | 8” | 10” | P-value | Portrait | Landscape | P-value3 | TL | TR | BL | BR | P-value | Horizontal | Vertical | P-value | Normal-in | Reverse-out | |
| Wrist | |||||||||||||||||
| Extension(°) | 0.01 | 12(2) | 14 (2) | 0.55 | 12(2) | 12(2) | <0.001 | 14 (2) A | 11 (2) B | 13 (2) A | 11 (2) B | 0.15 | 12(2) | 13(2) | 0.08 | 13(2) | 12(2) |
| Ulnar Deviation(°) | 0.09 | 15 (2) | 16 (2) | 0.03 | 14 (2) | 16 (2) | <0.001 | 18 (2) A | 16 (2) B | 15 (2) B | 13 (2) C | 0.12 | 16 (2) | 15 (2) | 0.2 | 14 (2) | 15 (2) |
| CMC | |||||||||||||||||
| Extension(°) | 0.8 | 6 (3) | 6 (3) | 0.91 | 6 (3) | 6 (3) | 0.28 | 7 (3) | 7 (3) | 6 (3) | 6 (3) | 0.77 | 6 (3) | 6 (3) | 0.74 | 6 (3) | 6 (3) |
| Abduction(°) | 0.32 | 3 (2) | 4 (2) | 0.22 | 3 (2) | 4 (2) | <0.001 | 7 (2) B | 0 (2) C | 10 (2)A | 0 (2) C | <0.001 | 4 (2) | 0 (2) | 0.19 | 4 (2) | 3 (2) |
| Pronation(°) | 0.78 | 1 (2) | 1 (2) | 0.89 | 2 (2) | 1 (2) | 0.03 | 0 (2) B | 1 (2) B | 1 (2) B | 3 (2) A | 0.64 | 1 (2) | 2 (2) | 0.49 | 1 (2) | 2 (2) |
| MCP | |||||||||||||||||
| Flexion(°) | 0.44 | 17 (2) | 17 (2) | 0.18 | 18 (2) | 17 (2) | <0.001 | 12 (2)C | 14 (2)BC | 16 (2)B | 20 (2)A | <0.001 | 18 (2) | 15 (2) | 0.11 | 17 (2) | 16 (2) |
| Abduction(°) | 0.65 | 14 (2) | 14 (2) | 0.58 | 15 (2) | 14 (2) | <0.001 | 16 (2)A | 11 (2)B | 17 (2)A | 10 (2)B | <0.001 | 15 (2) | 12 (2) | 0.24 | 16 (2) | 14 (2) |
| IP | |||||||||||||||||
| Flexion (°) | 0.23 | 31 (4) | 29 (4) | 0.08 | 30 (4) | 29 (4) | <0.001 | 6 (4)D | 25 (4)B | 17 (4)C | 40 (4)A | 0.07 | 31 (4) | 28 (4) | 0.13 | 31 (4) | 30 (4) |
Repeated Measures ANOVA with participant as a random variable, Tablet size (2 levels), Tablet Orientation (2 levels), Swipe Location (4 levels), Swipe Orientation (2 levels), Swipe Direction (2 levels) as fixed effect. Bold values indicate a significant effect (p<0.05).
TL= top left, TR=top right, BL=bottom left, BR=bottom right
For significant main effects, Tukey’s Post-Hoc groupings are ranked such that A>B>C. Values with the same superscript letters indicate no significant difference.
Table 3b.
Thumb and wrist movement range (90th %ile – 10th %ile): Across participant marginal means (and standard errors) for joint angles (°). Joint angles were expressed relative to a reference posture where the forearm, hand, and fingers were aligned along the longitudinal axis, and the thumb was extended and straightened to the lateral side of the index finger.
| Tablet Size | Tablet Orientation | Swipe Zone2 | Swipe Orientation | Swipe Direction | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| P-value | 8” | 10” | P-value | Portrait | Landscape | P-value3 | TL | TR | BL | BR | P-value | Horizontal | Vertical | P-value | Normal-in | Reverse-out | |
| Wrist | |||||||||||||||||
| Extension(°) | 0.27 | 10 (1) | 10 (1) | 0.51 | 10(1) | 11(1) | <0.001 | 13 (1) A | 9 (1) B | 13 (1) A | 8 (1) B | 0.12 | 10(1) | 11(1) | 0.21 | 10(1) | 10(1) |
| Ulnar Deviation (°) | 0.79 | 9 (1) | 10 (1) | 0.83 | 10 (1) | 10 (1) | 0.05 | 12 (1) A | 10 (1) B | 12 (1) A | 9 (1) B | 0.28 | 10 (1) | 10 (1) | 0.36 | 10 (1) | 11 (1) |
| CMC | |||||||||||||||||
| Extension(°) | 0.4 | 12 (1) | 12 (1) | 0.27 | 12 (1) | 13 (1) | 0.08 | 13 (1) | 12 (1) | 13 (1) | 11 (1) | 0.11 | 14 (1) | 12 (1) | 0.74 | 12 (1) | 12 (1) |
| Abduction(°) | 0.77 | 13 (2) | 14 (2) | 0.58 | 12 (2) | 14 (2) | 0.022 | 15 (2) A | 10 (2) C | 13 (2)B | 10 (2) C | <0.001 | 15 (2) | 10 (2) | 0.12 | 14 (2) | 13 (2) |
| Pronation(°) | 0.52 | 15 (2) | 14 (2) | 0.39 | 14 (2) | 14 (2) | 0.07 | 16 (2) | 14 (2) | 15 (2) | 14 (2) | 0.22 | 16 (2) | 15 (2) | 0.69 | 15 (2) | 15 (2) |
| MCP | |||||||||||||||||
| Flexion(°) | 0.44 | 17 (2) | 17 (2) | 0.18 | 18 (2) | 17 (2) | 0.14 | 18 (2) | 18 (2) | 19 (2) | 20 (2) | <0.001 | 16 (2) | 20 (2) | 0.18 | 18 (2) | 19 (2) |
| Abduction(°) | 0.45 | 16 (2) | 15 (2) | 0.58 | 15 (2) | 14 (2) | 0.32 | 17 (2) | 16 (2) | 17 (2) | 15 (2) | <0.001 | 16 (2) | 10 (2) | 0.77 | 16 (2) | 15 (2) |
| IP | |||||||||||||||||
| Flexion (°) | 0.17 | 14 (2) | 15 (2) | 0.74 | 15 (2) | 15 (2) | <0.001 | 20 (2)A | 13 (2)B | 18 (2)A | 12 (2)B | <0.001 | 13 (2) | 18 (2) | 0.51 | 14 (2) | 15 (2) |
Repeated Measures ANOVA with participant as a random variable, Tablet size (2 levels), Tablet Orientation (2 levels), Swipe Location (4 levels), Swipe Orientation (2 levels), Swipe Direction (2 levels) as fixed effect. Bold values indicate a significant effect (p<0.05).
TL= top left, TR=top right, BL=bottom left, BR=bottom right
For significant main effects, Tukey’s Post-Hoc groupings are ranked such that A>B>C. Values with the same superscript letters indicate no significant difference.
For the wrist, the top left swipe location had the greatest ulnar deviation (18°) and extension (14°) compared to the other three locations, while the bottom right location had the smallest (Table 3a). Similar to the thumb, the required wrist range of movement was also greater in the top left and bottom left swipe zones compared to the top right and bottom right swipe zones (Table 3b). The 10” tablet was associated with greater wrist extension (14°) compared to the 8” (12°). Landscape tablet orientation had greater ulnar deviation (16°) compared to portrait orientation (14°) (Table3a).
Muscle Activity
For median muscle activity, all measured forearm muscles except for flexor carpi radialis had significant differences across swipe locations (p<0.0001). For all the measured muscles, including extensor digitorum, extensor carpi radialis, extensor carpi ulnaris, extensor pollicis brevis, abductor pollicis longus, and flexor carpi ulnaris, the top left swipe location had the greatest muscle activity while the bottom right swipe location had the least (Table 4). Vertical swipe was associated with smaller extensor pollicis brevis muscle activity than horizontal swipe (p=0.0086). Interactions between swipe location and swipe orientation were significant for extensor carpi radialis, extensor pollicis brevis, and flexor carpi ulnaris (Figure 2).
Table 4.
Median Muscle Activity: Across participant marginal means (and standard errors) for surface electromyography (%MVC). RMANOVA Tablet Size, Tablet Orientation, Swipe Direction, Swipe Zone, Swipe Orientation
| Tablet Size | Tablet Orientation | Swipe Zone2 | Swipe Orientation | Swipe Direction | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Median EMG activity (% MVC) | P-Value1 | 10” | 8” | P-Value | Landscape | Portrait | P-Value3 | TL | TR | BL | BR | P-Value | Horizontal | Vertical | P-Value | Normal-in | Reverse-out |
| Extensor Carpi Ulnaris | <0.001 | 7.9(.9) | 6.7(.9) | <0.001 | 6.8(.9) | 7.7(.9) | <0.001 | 10.2(.9)A | 7.0(.9)B | 7.2(.9)B | 4.7(.9)C | 0.121 | 7.5(.9) | 7.1(.9) | <0.001 | 6.4(.9) | 8.2(.9) |
| Extensor Carpi Radialis | <0.001 | 5.6(.8) | 4.7(.8) | 0.010 | 5.0 (.8) | 5.2 (.8) | <0.001 | 5.8(.8)A | 4.9(.8)C | 5.4(.8)B | 4.4(.8)D | 0.408 | 5.2(.8) | 5.1(.8) | <0.001 | 4.9(.8) | 5.3(.8) |
| Flexor Carpi Ulnaris | <0.001 | 5.1(.8) | 4.3(.8) | 0.421 | 4.7(.8) | 4.7(.8) | <0.001 | 5.6(.8) A | 4.3(.8)C | 4.9(.8)B | 4.0(.8)C | 0.663 | 4.7(.7) | 4.7(.8) | <0.001 | 4.4(.8) | 5.0(.8) |
| Flexor Carpi Radialis | <0.001 | 7.4(1.2) | 6.3(1.2) | 0.815 | 6.8(1.2) | 6.8(1.2) | 0.357 | 6.6(1.2) | 6.9(1.2) | 7.0(1.2) | 6.8(1.2) | 0.658 | 6.9(1.2) | 6.8(1.2) | 0.777 | 6.9(1.2) | 6.8(1.2) |
| Abductor Pollicis Longus | <0.001 | 7.1(1.2) | 6.2(1.2) | 0.467 | 6.6(1.2) | 6.7(1.2) | <0.001 | 8.4(1.2)A | 6.1(1.2)C | 6.9(1.2)B | 5.2(1.2)D | 0.386 | 6.7(1.2) | 6.6(1.2) | <0.001 | 6.2(1.2) | 7.1(1.2) |
| Extensor Pollicis Brevis | 0.512 | 10.4(1.5) | 10.3(1.5) | 0.068 | 10.5(1.5) | 10.2(1.5) | <0.001 | 12.1(1.5)A | 10.0(1.5)B | 10.6(1.5)B | 8.8(1.5)C | 0.009 | 10.6(1.5) | 10.1(1.5) | <0.001 | 9.7(1.5) | 11.1(1.5) |
Repeated Measures ANOVA with participant as a random variable, Tablet Size (2 levels), Tablet Orientation (2 levels), Swipe Location (4 levels), Swipe Orientation (2 levels). Bold values indicate a significant effect (p<0.05).
TL= top left, TR=top right, BL=bottom left, BR=bottom right
For significant main effects, Tukey’s Post-Hoc groupings are ranked such that A>B>C>D. Values with the same superscript letters indicate no significant difference
Figure 2:
EMG plot for significant interactions (p<0.05) between swipe location and swipe orientation for flexor carpi ulnaris (FCU), extensor pollicis brevis (EPB), and extensor carpi radialis (ECR). Marginal means for median %MVC.
The 10” tablet was associated with greater median muscle activity compared to the 8” tablet for all measured muscles except for extensor pollicis brevis. Portrait tablet orientation was associated with greater extensor carpi radialis (p=0.01) and extensor carpi ulnaris (p<0.0001) muscle activity. Outward swipe action was associated with greater forearm muscle activity for all measured muscles except for flexor carpi radialis, compared to inward swipe action (p<0.0001). Other two-way or higher level interactions were not significant.
Performance
Completion time for across participant marginal means (1170 (55) ms) for the 5 repeated gestures in the top left swipe location was significantly greater (p<0.0001) than those in the bottom right swipe location (862 (55) ms). Participants also completed tasks more slowly (p<0.0001) when gestures were in the outward direction (1050 (53) ms) compared to inward (918 (53) ms).
User Perception
Gestures done further from the palm were associated with greater (p<0.0001) across participant marginal mean VAS discomfort scores (2.5 (0.3) cm) compared to closer to the palm (0.98 (0.3) cm). Actions in the horizontal direction also were associated with greater (p<0.0001) discomfort scores (2.0 (0.3) cm) compared to the vertical direction (1.5 (0.3) cm).
Discussion
The aim of this study was to determine how tablet form factors and swiping gesture designs affect thumb swiping performance, thumb posture, forearm muscle activity, and self-reported discomfort for a two-handed tablet grip. Overall, we saw effects of each of the independent variables (tablet size, tablet orientation, gesture location, gesture direction, and gesture orientation) on many of the thumb and wrist posture variables as well as the forearm muscles that articulate the joints of the wrist and thumb.
While thumb tapping and swiping are different motions, our results were similar across the two gestures in relationship to the independent variables (Trudeau, Catalano, Jindrich, & Dennerlein, 2013). The same biomechanical effort is required to reach and contact targets, though swiping requires maintained contact in a constrained time period. This maintained contact could lead to a longer duration of muscle activity and extreme joint postures which could be associated with MSDs and pain.
Consistent with our two hypotheses and our previous work on tapping, swipes starting in the top left location required significantly more extension and abduction across users’ thumb joints, and it also required much greater forearm muscle activation, compared to starting in the bottom right swipe location. In addition, the top left location resulted in the longest completion time for swipes, while the bottom right location resulted in the shortest completion times. Users’ wrist and thumb postures were closely related to the swipe locations on the device. Swipe locations closer to the center of the tablet required users to exert greater wrist extension and ulnar deviation, greater thumb CMC and MCP abduction, and greater IP extension to reach the target with the thumb. These results correspond with previous research on thumb reach during tapping tasks (Karlson, Robertson, Robbins, Czerwinski, & Smith, 2006; Trudeau, Udtamadilok, Karlson, & Dennerlein, 2012; Trudeau, Catalano, Jindrich, & Dennerlein, 2013).
With such an extreme thumb posture while reaching to the central tablet area, the associated forearm muscle activation was also elevated across all muscles except for flexor carpi radialis (Table 4). In contrast, the bottom right swipe location, because of its proximity to the palm, allowed users to swipe with the smallest wrist joint angles and left the thumb MCP and IP joints more flexed (Trudeau, Catalano, Jindrich, & Dennerlein, 2013), which in turn also resulted in the smallest muscle activity across the forearm muscles measured.
These study results serve as evidence that certain swipe orientations (vertical or horizontal) may be easier for users in certain locations of the tablet screen. Both vertical and horizontal swipes showed similar trends across muscles where the top left swipe location required the greatest muscle activity and the bottom right swipe location required the least. However, the increase in muscle activity from bottom right swipe location to the top left location was much more drastic for vertical swipes than horizontal swipes. Specifically, vertical thumb swipes in a two-handed grip can be utilized when the target is closer to the palm, but should be avoided as the target gets further away towards the geometric center of the tablet.
Compared to the 8” tablet, the 10” tablet required greater muscle load across all muscles measured to support the extra weight, but overall the tablet size had little impact on thumb posture or completion time. The target design and reach requirements were the same on both devices, so this difference in muscle activation may be due to the weight difference between the two tablets. Because the center of mass is farther from the supporting hand, creating a larger moment when holding the larger tablet, which may require more forearm muscle activity. The smaller muscle activity associated with the smaller tablet were consistent with Pereira et al.’s research that found smaller muscle activity of the tablet holding arm were associated with smaller tablets (Pereira, Miller, Huang, Odell, & Rempel, 2013).
Consistent with Trudeau et al., which found two-handed grip on the tablet bottom requires an increased moment arm with the hand supporting location being further away from the tablet center of mass, greater muscle activity was associated with portrait orientation than landscape orientation (Trudeau, Catalano, Jindrich, & Dennerlein, 2013). Holding the tablet in the portrait orientation further increases the moment arm as compared to the landscape orientation.
This study found that swipe location most affected upper limb biomechanics. These results imply that while manufacturers strive to improve tablet hardware form factors, software interface design can also significantly improve tablet user experience. By avoiding thumb reaching into the center area during a two-handed grip, users will be able to operate the tablet with a more relaxed thumb and wrist posture, along with lower activity across forearm muscles.
The current study results need to be considered within the context of the study limitations. First, we only considered the two-handed grip support condition with only the thumb used for swipes. Hence, other support conditions such one-handed grip with index finger swiping or with the device supported on a desk, with or without a case, could potentially yield different results. This experiment had a low sample size, and thus may not be fully generalizable to the larger population. The small values found for the posture and muscle activity may not correlate with increased risk of MSDs clinically. Another limitation is that this was a laboratory study with a short duration of swiping tasks. Real life tablet swiping tasks could have different force requirements, durations, frequencies, and psychological stress levels that could impact users’ biomechanical loads. A next step following this study would be to compare the results from this study with different support conditions while interacting with other fingers. Longer tasks that better simulate real life office computer work with added psychological stress would also be appropriate.
The study found that the 10” tablet and the portrait orientation required greater muscle activity compared to the 8” tablet and the landscape orientation, respectively. However, due to the tablet app design and time constraint, we chose a fixed dimension when defining swipe locations. Future studies should consider varying the swipe location sizes based on different tablet size and orientation. Such information can provide insights regarding when swipe locations cease to matter for a specific tablet size and orientation.
Conclusion
The study results demonstrated that, for a two-handed grip on a tablet, thumb swiping time, thumb/wrist posture, forearm muscle activity, and self-reported discomfort vary across swipe locations. Swipe location closest to the palm allows users to swipe with a more neutral thumb and wrist posture and requires less forearm muscle activity. As swipe location gets further away towards the center of the tablet, increased forearm muscle activity and greater thumb extension and abduction along with greater wrist extension and adduction are required to reach the target. Users had the best performance when the swipe location was closest to their palm and they reported lower discomfort levels when swipe location was closer to their palm. These results should inform tablet hardware and software manufacturers to design tablet computer interfaces that allow improved performance and usability while inducing a more neutral thumb and wrist posture with smaller muscle load demands.
Acknowledgments
This research was supported in part by the National Science Foundation (0964220), The Office Ergonomics Research Committee, and the National Institute for Occupational Safety and Health (T42 OH008416). We would like to thank Mark Janelli from Northeastern University for contributions to study design, experiment preparation, data collection and compilation and Philippe Dixon and Aubrie Sanchez for help reviewing this manuscript.
References
- Accot J, & Zhai S (1997). Beyond Fitts’ law: models for trajectory-based HCI tasks CHI EA ‘97 CHI ‘97 Extended Abstracts on Human Factors in Computing Systems (p. 250). Atlanta: ACM. [Google Scholar]
- Asundi K, Odell D, Luce A, & Dennerlein J (2012). Changes in posture through the use of simple inclines with notebook computers placed on a standard desk. Ergonomics, 43(2), 400–7. [DOI] [PubMed] [Google Scholar]
- Bergstrom-Lehtovirta J, & Oulasvirta A (2014). Modeling the functional area of the thumb on mobile touchscreen surfaces CHI ‘14 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 1991–2000). Toronto: ACM. [Google Scholar]
- Billinghurst SS, & Vu KL (2015). Touch screen gestures for web browsing tasks. Computers in Human Behavior, 53, 71–81. [Google Scholar]
- Dennerlein JT, Martin DB, & Hasser C (2000). Force-feedback improves performance for steering and combined steering-targeting tasks CHI ‘00: Proceedings of the SIGCHI conference on Human Factors in Computing Systems (pp. 423–9). Th Hague: ACM. [Google Scholar]
- Dennerlein J, & Johnson P (2006). Changes in upper extremity biomechanics across different mouse positions in a computer workstation. International Journal of Industrial Ergonomics, 49(14), 1456–69. [DOI] [PubMed] [Google Scholar]
- Gustafsson E, Johnson PW, & Hagberg M (2010). Thumb postures and physical loads during mobile phone use – A comparisonof young adults with and without musculoskeletal symptoms. Journal of Electromyography and Kinesiology, 20(1), 127–35. [DOI] [PubMed] [Google Scholar]
- Gustafsson E, Johnson PW, Lindegard A, & Hagberg M (2011). Technique, muscle activity and kinematic differences in young adults texting on mobile phones. Ergonomics, 54(5), 477–87. [DOI] [PubMed] [Google Scholar]
- Hill S (2013, October 11). History of the phablet. Retrieved from Android Authority: http://www.androidauthority.com/phablet-history-279494/
- Jeong H, & Liu Y (2017). Effects of touchscreen gesture’s type and direction on finger-touch input performance and subjective ratings. Ergonomics, 60(11), 1528–1539. [DOI] [PubMed] [Google Scholar]
- Karlson AK, Robertson GG, Robbins DC, Czerwinski MP, & Smith GR (2006). FaThumb: A Facet-Based Interface for Mobile Search CHI (pp. 711–720). Montreal: ACM Press. [Google Scholar]
- Ko PH, Hwang YH, & Liang HW (2016). Influence of smartphone use styles on typing performance and biomechanical exposure. Ergonomics, 59(6), 821–8. [DOI] [PubMed] [Google Scholar]
- Lin M, Young J, & Dennerlein J (2015). Evaluating the effect of four different pointing device designs on upper extremity posture and muscle activity during mousing tasks. Applied Ergonomics, 47, 259–264. [DOI] [PubMed] [Google Scholar]
- Odell D, & Chandrasekaran V (2012). Enabling comfortable thumb interaction in tablet computers: a Windows 8 case study. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, (pp. 1907–1911). [Google Scholar]
- Park YS, & Han SH (2010). One-handed thumb interaction of mobile devices from the input accuracy perspective. International Journal of Industrial Ergonomics, 40(6), 746–56. [Google Scholar]
- Pereira A, Miller T, Huang Y, Odell D, & Rempel D (2013). Holding a tablet computer with one hand: effect of tablet design features on biomechanics and subjective usability among users with small hands. Ergonomics, 56(9), 1363–75. [DOI] [PubMed] [Google Scholar]
- Perotto A (1994). Anatomical Guide for the Electromyographer: The Limbs and Trunk. Springfield, IL: Charles C Thomas. [Google Scholar]
- Trudeau M, Asakawa DS, Jindrich D, & Dennerlein JT (2016). Two-handed grip on a mobile phone affords greater thumb motor performance, decreased variability, and a more extended thumb posture than a one-handed grip. Applied Ergonomics, 52, 24–8. [DOI] [PubMed] [Google Scholar]
- Trudeau M, Catalano P, Jindrich J, & Dennerlein J (2013). Tablet Keyboard Configuration Affects Performance, Discomfort and Task Difficulty for Thumb Typing in a Two-Handed Grip. PLoS One, 8(6). [DOI] [PMC free article] [PubMed] [Google Scholar]
- Trudeau M, Udtamadilok T, Karlson A, & Dennerlein J (2012). Thumb motor performance varies by movement orientation, direction, and device size during single-handed mobile phone use. Human Factors, 54(1), 52–9. [DOI] [PubMed] [Google Scholar]
- Trudeau M, Young J, Jindrich D, & Dennerlein J (2012). Thumb motor performance varies with thumb and wrist posture during single-handed mobile phone use. Journal of Biomechanics, 45(14), 2349–54. [DOI] [PubMed] [Google Scholar]
- Villamor C, Willis D, & Wroblewski L (2010, April 15). Touch Gesture Reference Guide. Retrieved from https://static.lukew.com/TouchGestureGuide.pdf
- Winter D (2005). Biomechanics and Motor Control of Human Movement- Third Edition.
- Young J, Trudeau M, Odell D, Marinelli K, & Dennerlein J (2012). Touch-screen tablet user configurations and case-supported tilt affect head and neck flexion angles. Work, 41(1), 81–91. [DOI] [PubMed] [Google Scholar]
- Young J, Trudeau M, Odell D, Marinelli K, & Dennerlein J (2013). Wrist and shoulder posture and muscle activity during touch-screen tablet use: effects of usage configuration, tablet type, and interacting hand. Work, 45(1), 59–71. [DOI] [PubMed] [Google Scholar]