BackSwipe: Back-of-device Word-Gesture Interaction on Smartphones

Abstract

Back-of-device interaction is a promising approach to interacting on smartphones. In this paper, we create a back-of-device command and text input technique called BackSwipe, which allows a user to hold a smartphone with one hand, and use the index finger of the same hand to draw a word-gesture anywhere at the back of the smartphone to enter commands and text. To support BackSwipe, we propose a back-of-device word-gesture decoding algorithm which infers the keyboard location from back-of-device gestures, and adjusts the keyboard size to suit the gesture scales; the inferred keyboard is then fed back into the system for decoding. Our user study shows BackSwipe is feasible and a promising input method, especially for command input in the one-hand holding posture: users can enter commands at an average accuracy of 92% with a speed of 5.32 seconds/command. The text entry performance varies across users. The average speed is 9.58 WPM with some users at 18.83 WPM; the average word error rate is 11.04% with some users at 2.85%. Overall, BackSwipe complements the extant smartphone interaction by leveraging the back of the device as a gestural input surface.

1. INTRODUCTION

One-handed Interaction with a smartphone via the surface on its back (Back-of-device input) has been demonstrated as a promising approach to address some of the well-known problems that are associated with touch input using the thumb, such as “fat finger” or reachability [23, 24]. With the recent development of new smartphone technologies, especially the ones offering a rear display (e.g., Samsung foldable phones), sensing back-of-device input is becoming increasingly practical. However, the bottleneck lies in the software as the existing back-of-device input is largely limited to simple directional strokes or tapping [11, 49, 55]. The full potential of this input modality on a smartphone is yet to be explored.

In this paper, we investigate how word-gesture input (a.k.a ShapeWriting, Swyping, or gesture typing input) [26, 60] can be carried out on an imaginary keyboard to trigger commands or perform text input via the back of a smartphone (Figure 1). Performing word-gesture input on the back of the device has several unique benefits. First, the technique is well suited for one-handed input scenarios as it allows a user to interact with the smartphone with the index finger that is free from gripping the device with one hand (Figure 1). Second, the user can input words or shortcut commands via continuous finger gestures. This is more expressive than simple directional strokes, such as swiping left or right. Third, since user input is carried out on the back of the device, it does not conflict with the system-level gestures performed on the front of the device (e.g., scrolling and swiping).

Figure 1:

Demonstration of BackSwipe. The user is holding the smartphone with one hand, and uses the index finger to input the word-gesture of the command “copy” on the back of the device. The command is triggered on the composing text.

Despite all these benefits, word-gesture input via the back of the device is challenging because the motion of the index finger is largely restricted by the hand holding the device, causing difficulty for the user to remain accurate in positioning the fingertip on the keyboard. Additionally, the input finger is obscured by the device, which could make the position control more difficult. These factors could result in gestures that are inaccurate and hard to be recognized. A decoder (e.g., SHARK² decoder [26, 60]) can potentially solve the problem but it requires the knowledge of the location and size of the imaginary keyboard, which is unknown as they differ among people with different hand sizes and holding postures.

To address these challenges, we designed and implemented Back-Swipe, a one-handed input technique designed for back-of-device input on a smartphone. BackSwipe infers the location of the imaginary keyboard based on a user’s input gestures to create a decoder for back-of-device word-gesture decoding. We have applied this decoder to support both command input and text entry on smart-phones.

BackSwipe has the following advantages over extant front-screen one-handed interaction techniques. As a command input method, BackSwipe avoids conflict with existing front-screen gestures. It provides an additional channel for inputting commands and short text which can co-exist with extant front-screen gestures with-out consuming the affordance of the front screen. As a text entry method, BackSwipe avoids visual interferences with the content displayed on the front screen because there is no need to constantly show a keyboard. It also frees the entire front-screen for interaction with the displayed digital content: Users can freely interact with the digital content on the front in the middle of the text entry. It minimizes the visual and interaction interference on the front screen by providing a dedicated input space on the back for command and text input.

Our user research shows BackSwipe is feasible and promising, especially when a user needs to enter commands or short text in the one-hand holding posture. Our user study shows users can enter commands at an average accuracy of 92% with the speed of 5.32 seconds/command. The text entry performance varies across users. The average speed is 9.58 WPM with some users at 18.83 WPM. The average word error rate is 11.04% with some users at 2.85%.

2. RELATED WORK

Our work is related to the research in word-gesture input systems and back-of-device input.

2.1. Word-Gesture Input for Text and Command Input

Word-gesture typing allows users to enter text using a continuous gesture stroke to traverse the letters on the keyboard. It was first introduced by Zhai and Kristensson [26, 59–61] and has been widely adopted by various commercial keyboards.

Researchers have explored word-gesture typing on a variety of input devices. For example, the bimanual gesture keyboard [6] enabled users to use both hands and multiple strokes to enter one word. i’sFree [67] supported word-gesture input in an eyes-free manner with a touchpad. Chen et al. [8] explored text entry in VR with pressure-sensitive touchscreen devices. These techniques rely on a touchscreen or a touch surface for input, but did not explore other input spaces. Other than using the finger or a stylus, researchers have also extended word-gesture input using head or hand movement. For example, Vulture [40] investigated mid-air word-gesture input by tracking users’ hand movement projections on a display. Yu et al. [58] used head rotation for gesture typing in HMDs. Yeo et al. proposed SWiM [56], a tilt-based system to allow single-handed word-gesture input on a smartphone. RotoSwype [19] proposed a ring-based text entry technique that uses the orientation of the ring to support typing. Our work is particularly related to eyes-free gesture typing [67] which also inferred keyboard location for gesture typing. The main difference is that in [67] the keyboard size and x position are known, while for back-of-device input keyboard size, both x and y locations are unknown, which added more complexity for decoding.

In addition to text input, gestures have also been used to trigger commands. Many of the early work in this space, such as marking menus [28–30] and its adaptations, such as hierarchical marking menus [63], wave menus [2], flower menus [3], wavelet menus [16], MarkPad [17], imaginary gestures [20, 21], and M3 gesture menu [64], requires the user to memorize the mapping between a command and its corresponding gesture, thus making it hard for novice users to adopt. Some of the recent research addresses this issue by assigning semantic meanings to the gestures or using user-defined gesture shortcuts [32, 34–36, 43, 50]. In contrast, word-gesture commands have the benefit that the shape of the gestures directly reflects the name of the corresponding command, thus are easier to memorize and use [1, 12, 27, 65].

Built on the existing word-gesture input techniques and their variants, we explored a new venue of research to investigate how to support word-gesture input on the back of a smartphone (i.e., BackSwipe). Different from the existing word-gesture input methods which require a keyboard to be shown to the user on the front ( e.g., SWiM [56]) or touch input to be carried out on the front screen (e.g., one-handed front-screen gesture typing [26, 60]), BackSwipe minimizes the visual and interaction interference on the front screen by providing a dedicated input space on the back for command and text input.

2.2. Back-of-device Interaction on Smartphones

Back-of-device interaction was initially proposed to solve the “fat finger” problem, i.e., direct input on the front of the device may have finger occluding the targets, especially when they are smaller than the finger width [23, 24]. To address this problem, researchers proposed to use additional hardware, such as attaching physical buttons [25, 31, 47] or tactile landmarks [10] on the backside of the device for text entry or triggering applications (e.g., personal calendars). Others explored back-of-device interaction on two-sided touch surfaces [7, 11, 14, 22, 49, 53, 54, 62]. For example, Hybrid-Touch [49] allows the user to interact with a handheld device by simultaneously touching the front display using a stylus and the trackpad on the back using the index finger holding the device. LensGesture [55] detects finger gestures on the back of a smart-phone using the device’s back-facing camera. XSide [14] allows the user to enter stroke-based passwords via the front or back of the device to protect against shoulder surfing. BackXPress [11] uses pressure sensors on the back of the device to allow the user to switch between different input modes.

Despite all the benefits of back-of-device input (e.g., eliminating the “fat finger” problem, extended input space), a significant trade-off is the occlusion of finger movement. LucidTouch [52] mitigates this problem by using a pseudo-transparency display showing the finger movement on the device’s back. NanoTouch [4] is similar except that the device simulates the see-through effect using an image of the finger touching the backside of an ultra-small wearable device.

Existing back-of-device interaction techniques are mostly performed through simple directional strokes (such as swiping up, down, left, or right), or tapping (on physical buttons or different regions of the device). Shimon et al. [48] performed an elicitation study to understand how users map these gestures to smartphone commands. In this work, we extended the gesture space by exploring how to apply word-gesture command input to back-of-device interaction.

3. EXPERIMENT 1: UNDERSTANDING BACK-OF-DEVICE WORD-GESTURE INPUT

We first carried out a study to understand how users performed word-gesture input on the back of a smartphone. Such an under-standing would guide us in hldesigning a decoder to support back-of-device word-gesture input.

3.1. Design and Tasks

We conducted a Wizard-of-Oz experiment to collect the participants’ gestures on the back of a smartphone. Participants were instructed to hold the phone naturally, imaging an invisible keyboard on the back of the phone which could correctly decode their input. We used a ZTE AXON M as the test device. The device has two 5.2-inch screens, acting as one front and one back screen. In the experiment, the front screen was designed to show prompts, and the back screen was used to collect the gesture input data.

The study adopted a text transcription task. A short phrase was displayed on the front screen and the participant was instructed to draw a word-gesture on the back of the device to enter the phrase. A short line was displayed under the current target word in the phrase and would advance to the next word after the word-gesture for the current word was drawn. The participants were instructed to perform word-gesture input to transcribe text with the index finger on the back screen while holding the phone with the same hand. The gesture traces was displayed on the phone’s front screen to provide feedback, as shown in Figure 2. By default, the keyboard layout was not shown on the screen. We adopted such a design because there is evidence showing that many users are able to input text on an imaginary keyboard on the phone [66], remote control [67], or hand-held touchpad [37], given the dominance and users’ familiarity with Qwerty layout. In case a user could not recall the location of a particular key, keeping the index finger still for 300ms at the back screen would bring up a Qwerty layout on the front screen. The keyboard would disappear once the user took the index finger off the screen upon the completion of the current word. We instructed the participant to assume that there existed an imaginary keyboard at the back of the screen that would successfully decode the word-gesture she entered.

Figure 2:

(a) A participant is entering a word with the index finger on the back of the phone. (b) a screenshot of the front screen. The displayed gesture on the front screen shows the finger trace as if the user is looking through the phone.

The testing phrases were selected from a subset of the MacKenzie and Soukoreff phrase set [39, 57]. The same test set was used for all participants. The participants were required to complete 4 blocks in the experiment and could take a short break after completing each block. Each block contains 10 phrases whose orders were randomized. Before the formal study, we instructed the users to complete a 5 minutes warm-up to get familiar with word-gesture input on the back of the device.

3.2. Participants and Apparatus

Ten participants (3 females, all right-handed) participated in the study. The ages of the users were from 25 to 30 (M = 28). Their median familiarity with the Qwerty layout was 4.5 (1: very unfamiliar; 5: very familiar). Their median familiarity with gesture typing was 3. All of the participants were instructed to use their preferred posture to hold the phone during the experiment.

A ZTE AXON M dual-screen foldable mobile phone (Qualcomm MSM8996 Snapdragon processor, Quad-core CPU with a 2×2.15 GHz Kryo and a 2×1.6 GHz Kryo, Adreno 530 GPU, 4GB RAM, 64GB internal storage, dual 5.2-inch 1080×1920 screen) was used in the study. In the experiment, the front screen was used to display the trial information and other visual guidance. The back screen was used to collect the gesture traces.

In total, the study included: 10 participants × 4 blocks × 10 phrases = 400 trials.

3.3. Results

To understand the positions of the imaginary keys and keyboards, we first inferred the key distribution from the collected gesture traces.

3.3.1. Inferring key positions.

Similar to the previous research [67], we used the dynamic time warping (DTW) algorithm [46] to infer the imagined key positions of a word w from its corresponding gesture input g. We first generated a gesture template t for word w. The template was created by connecting the centers of corresponding letters in w on a standard Qwerty layout. The Qwerty layout parameters were obtained from AOSP keyboard for a 1920×1080 resolution Android device. Then we sampled g and t into N (N = 300) equidistant points and applied an optimal match between g and t. In this way, for every letter c in w, we obtained its key center in the sampled template pattern t as t_c, and its corresponding point g_c in the sampled gesture g as the inferred key position. While applying the DTW algorithm to match g with t, we sampled the gesture based on the gesture trace coordinates instead of the timestamps, so the results were unlikely to be affected by the gesturing time between letters.

3.3.2. Distribution of the imaginary keys.

Figure 3a shows the distribution of the imaginary key positions inferred from the gestures. The y-direction variance is greater than the variance in the x-direction. The mean standard deviation of the key positions was 4.1 mm and 8.8 mm in x and y-directions. The imaginary keyboard center was mainly located in the upper right part on the back of the phone. We also show the imaginary key distributions of 7 randomly-picked participants in Figure 3b-h. The imaginary keyboard positions differ across users. We also calculated the input speed in the study. The average input speed was 21.9 (SD=6.9) words per minute (WPM).

Figure 3:

Imaginary keyboard positions and the inferred key distributions (95% confidence ellipses) for back-of-device word-gesture input. The borders illustrate the boundaries of the back screen of the smartphone. (a): the average distribution of all participants. (b) to (h): key distributions obtained from 7 participants randomly picked from all participants.

3.3.3. Discussion.

Our investigations led to the following findings, providing a guideline on how to design a decoder for back-of-device word-gesture input.

Users were capable of inputting gestures on the back of the device.

Figure 3a shows that the key centers’ relative positions on the imaginary keyboard are similar to those on a standard Qwerty layout. Participants were able to recall the Qwerty layout quite accurately, even when they were gesturing on the device’s back without the visual of fingers. The average input speed was around 22 WPM, which can be considered the upper limit for back-of-device word-gesture input: the participants were not interrupted by error corrections or candidate selections.

The gesture input behavior on the back of the device was different from regular gesture input.

There were significant differences between a back-of-device keyboard layout and the standard Qwerty layout. For instance, different users have varying preferences on where the imaginary keyboard position should be, as shown in Figure 3(b)-(h). The positions of imaginary keyboards differed across users. It is probably because the imaginary keyboard position is easily influenced by the users’ postures, which is further affected by their hand size and the coordination of finger movements. We also noticed that users tend to adjust their hand posture when they feel tired or uncomfortable during the study. This indicates that the imaginary keyboard should not be assigned to a fixed position as a regular smartphone keyboard. At the same time, the imaginary keyboard position only undergoes a small change within the same phrase. Assuming the participants entered every word correctly, we estimated the keyboard position of each gesture. The average keyboard location change in a phrase was 6.5 (SD=2.3) mm.

Another interesting finding is that across all the users, the shape of the imaginary keyboard is close to a “square” other than a rectangle: the width to height ratio of a standard Qwerty layout in an AOSP keyboard is 12 : 5 = 2.4, where the imaginary keyboard is 12 : 9 = 1.3

4. BACK-OF-DEVICE WORD-GESTURE DECODING

Based on the findings drawn from the Wizard-of-Oz study, we investigated how to design a decoder for back-of-screen word-gestures. The basic principle of word-gesture decoding [26, 60] is to combine the probability of an intended word estimated from the input gesture (a.k.a spatial probability c(w)) with the probability of estimated from the language context (a.k.a language probability l (w)) to obtain the overall probability of a word w being the intended input given an input gesture:

$$s\left(w\right)=\frac{l\left(w\right)c\left(w\right)}{{\sum }_{i\in W}l\left(i\right)c\left(i\right)},$$ (1)

where W is a lexicon or command set containing i words. We follow the same principle (Equation 1) for decoding back-of-device word-gestures, described as below.

4.1. Spatial Probability

We obtain the spatial probability of a word w by refining the classic SHARK ² decoding algorithm [26, 60], which was originally designed to decode word-gestures drawn on a visible soft keyboard. The core challenge of applying SHARK ² of decoding back-of-device word-gestures is that, in the back-of-device interaction, the size and location of the keyboard are unknown: the imaginary keyboard could be anywhere on the back of the device with any size, depending the holding position. Based on the finding that the keyboard position only undergoes a small change in the same sentence, we created the following method to infer the keyboard location from the input gesture.

We estimate the keyboard size (width and height) based on the data collected in Experiment 1. The study data showed although the keyboard center varies across users, both width and height of the keyboard underwent only small changes across users. The mean width (SD) was 23.9 (4.5) mm, and the mean height (SD) was 18.6 (5.5) mm. We therefore set the keyboard size to 23.9 ×18.6 mm.

Keyboard Position Estimator.

We have designed an iterative-updating algorithm to update the center of the imaginary soft keyboard after entering each word. Assuming that the keyboard center for entering the word w_{t −1} is c_{t −1}, the keyboard size is 23.9×18.6 mm, and keys are arranged according to a Qwerty layout, we can estimate the center of each key, referred to as a_{t −1}, b_{t −1}, …, z_{t −1}. After a user enters the w_t, we estimate the center of each letter in the word w_t using the DTW algorithm, as described in Section 3.3.1 of Experiment 1. We then calculate the position shift of the corresponding letter, compared it with its previous position for w_{t −1}. The shift of the keyboard center will be the average of all the letter shifts. For example, assuming a user draws a gesture to enter and. The iterative-updating algorithm first estimates the new letter centers a_t, n_t, and d_t using the previously described DTW algorithm, and calculates the letter positions shifts ∆a = a_t − a_{t −1}, ∆n = n_t − n_{t −1}, and ∆d = d_t − d_{t −1}. The keyboard position shift is calculated as $\overline{\Delta }=\frac{\Delta a+\Delta n+\Delta d}{3}$.

When entering the first word in a sentence, or a single command, we use only the keyboard size information for decoding since there is no prior knowledge on the keyboard center c. It means that we use only the shape channel of the SHARK ² decoder for decoding, because this channel does not require the information about the keyboard center for calculating the shape similarity between input gestures and word templates. After entering the first word, we then estimate the keyboard center position based on the input gesture, and treat it as c₁. We then apply the iterative-updating algorithm to update the keyboard location after each word, and feed it back into a SHARK² decoder [26] to decode subsequent gestures.

4.2. Language Probability

For text entry, we used the GPT [44] language model, pretrained by OpenAI. The vocabulary size of the model is 40478. For the first two words in a sentence which have little language context to leverage, we used a 2-gram language model (size: 7 million bigrams) which was trained over the Corpus of Contemporary American English (COCA) [13](2012 to 2017). The Corpus contains over 5 million sentence.

4.3. Decoder Architecture

The workflow of the algorithm is shown in Figure 4. The new component added on top of a regular gesture typing decoder is the keyboard position estimator, as described in the previous section. We view the BackSwipe decoding algorithm as an extension of the eyes-free decoder [67]: extending the algorithm from estimating the y position of the imaginary keyboard only (the keyboard vertical location learner in [67]) to estimating keyboard size, and both x and y position.

Figure 4:

The architecture of BackSwipe’s decoder.

4.4. BackSwipe for Text Entry and Command Input

We have applied the decoding algorithm to support both text entry and command input. During text entry, the system will display 4 suggestions, arranged in a 2×2 grid after each gesture input. A user then swipes at the back of the screen in one of the 4 directions to select the corresponding suggestion. In case the user cannot recall the key position on a Qwerty layout, she may hold the finger still on the back of the device for 0.3 seconds to invoke a Qwerty keyboard displayed on the front screen.

We have also applied BackSwipe to support command input, by replacing the lexicon with a set of commands. A user triggers a command by back-swiping (i.e., drawing a word-gesture at the back of the device) the corresponding command label. By default, the recognized command will be executed upon the input finger lifts off the back surface. If the recognition scores of the top two candidates are close enough (i.e., the score difference is less than 5% of the average score), the system will display 4 command candidates, and the user swipes on the back screen in the corresponding direction to trigger the command. Similar to text entry, holding the input finger still on the back for 0.3 seconds will trigger the display of a Qwerty layout on the front screen to help the user recall key positions.

We have implemented a BackSwipe decoder as described in this section on a ZTE AXON M dual-screen foldable phone which ran Android 7.1.2. The source code can be found at Bitbucket.¹ Next, we carried out two experiments to evaluate BackSwipe, for command input and text entry separately.

5. EXPERIMENT 2: EVALUATING BACKSWIPE FOR COMMAND INPUT

We carried out Experiment 2 to evaluate BackSwipe for command input.

5.1. Design

This experiment is a command activation task. The participants were required to complete multiple command activation trials using BackSwipe, among a set of 30 commands. The size of the command set was close or bigger than the size of commonly used command sets in a mobile application (e.g., a Chrome web browser supports 13 commands in its first-layer menu).

Before the experiment, the participants were shown 30 target commands and their corresponding stimulus icons on a sheet. The users were required to memorize at least 90% of the commands. Each command could be triggered by one of the two words. For example, to trigger the “clock” icon, the users could enter either “clock” or “timer”. The words were created from [15] by judging whether they reflected the meaning of the icon. The commands and corresponding words for triggering them are shown in Appendix (Table 2). Since this experiment was not a text composition or transcription task, we replaced the lexicon in the decoder with a set of 60 words (2 words per command).

5.2. Procedure

At the start of each trial, a stimulus icon was displayed at the upper part of the front screen. The participants were required to input the command by BackSwipe. After each gesture input, the decoder would analyze the input confidence and decide whether to provide command candidates following the policy described in Section 4.4, as shown in Figure 5. If candidates were provided, participants needed to select the intended command by swiping to the corresponding direction on the device’s back. If not, the first candidate command from the decoder was used as the default command output. A trial was completed if the input command was correct, or the participants made three consecutive failed attempts. Before the formal test, the participants were instructed to get familiar with the test procedure and BackSwipe in a warm-up session of 5 minutes. The commands in the warm-up session would not appear in the formal test.

Figure 5:

(a) A user is inputting the command “copy” with BackSwipe. (b) The stimulus icon is displayed on the front screen. (c) The decoder shows the command “spin” with four more candidate commands. Since “spin” is not the intended command, the participant swipes her finger to the direction of the intended word “copy” as if the finger is located at the center of the suggestion grid (i.e., top-right direction). In other words, drawing a gesture towards top-right corner would select the “copy” command

We randomly selected 12 out of 30 commands from the command set. Based on previous research, the command input patterns of launching applications [41] or selecting menu items [9, 33] usually follow certain distributions (e.g. Zipfian distribution). We generated the command occurrence frequency according to Zipfian distribution. The occurrences for the 12 commands was (7, 5, 4, 4, 2, 2, 1, 1, 1, 1, 1, 1). We counterbalanced the test commands to frequencies across all participants, ensuring that each command maps to each frequency the same number of times. The same set of commands was used across all participants. The order of the commands was randomized for each participant. The experiment was divided into 5 blocks, each consisting of 6 trials.

5.3. Participants and Apparatus

12 participants (3 females, all right-handed) from 25 to 29 years old took part in the study. The self-reported median familiarity (1: very not familiar; 5 very familiar) with word-gesture input and Qwerty layout were 3 and 4, respectively. We instructed the participants to complete the experiment with their preferred hand postures. The same ZTE AXON M device was used.

In total, the study included: 12 participants × 5 blocks × 6 phrases = 360 trials.

5.4. Results

Command Input Time.

This metric indicates how fast a user could trigger a command on the back of the device. It is measured as the elapsed time from the moment a participant starts drawing a gesture to the end of the trial. The average command input time was 5.32 (SD=1.28) seconds. To understand the performance of the participants as they progressed in the experiment, we show the mean (95% CI) command input time across the five blocks in Figure 6a. A repeated-measures ANOVA did not show a significant main effect of blocks on the command input time (F_{4, 44} = 0.86, p = 0.49).

Figure 6:

Average (95% CI) of command input time (in seconds) and command error rate (%) by test blocks.

Command Input Error Rate.

The command input error rate was the ratio of failed trials over the total number of trials. The average command input error rate among all participants was 7.50% (SD=6.05%). The mean (95% CI) command input error rate across the five blocks is shown in Figure 6b. A repeated-measures ANOVA did not show a significant main effect of blocks on error rate (F_{4, 44} = 0.23, p = 0.92).

Suggestion Usage.

This metric indicates how useful the command suggestions were in the experiment. It was calculated as the ratio of successful commands triggered using suggestions over the total number of successful trials. The average suggestion usage was 29.8% (SD=11.0%), indicating that for over 70% of successful command trials users did not need suggestions.

Reference Keyboard Usage

This metric measures how often the reference keyboard was invoked on the front screen during the experiment to guide the user input. It was calculated as the number of times the keyboard showed up on the screen over the total number of input gestures. The average reference keyboard usage was 0.43 (SD=0.05) times, indicating that 43% of times the reference keyboard was invoked.

Subjective Feedback.

At the end of the experiment, participants were required to provide a continuous numerical rating (1: least demanding, 10: most demanding) on the experiment’s mental and physical demands. Mental demand described how much mental effort was required to complete the study. Physical demand described how much physical effort was required. The average mental demand was 4.9 (SD=2.3). The average physical demand was 5.7 (SD=2.3). The participants were also asked to rate the BackSwipe according to their impression on a scale of 1 to 5 (1: dislike, 5: like). The average rating was 3.8 (SD=0.9).

5.5. Discussion

Our experiment led to the following findings.

First, BackSwipe is a promising method to input gesture command. The results showed that users could successfully enter commands at an average error rate of 7.5%, at the speed of 5.32 seconds per command. Four users (1/3 of the participants) entered commands at a speed faster than 4 seconds/command. It shows that BackSwipe can help users to conveniently enter command shortcuts in a one hand holding posture.

Second, suggestions are necessary for triggering commands with similar shapes, such as “cut” and “copy”. Entering these commands requires high input accuracy, which might lead to more errors. Participants also tend to make more errors in complicated gestures. For example, to input the command “delete”, users need to trace their fingers back and force around the letter “e”, and thus would make more mistakes. With the help of command suggestions, users could finish the task more easily are relaxed.

6. EXPERIMENT 3: EVALUATING BACKSWIPE FOR TEXT ENTRY

After evaluating BackSwipe for command input, we carried out Experiment 3 to evaluate it for text entry.

6.1. Design and Tasks

This study was a text transcription task. Similar to the design in Experiment 1, the participants were instructed to transcribe the phrases shown on the front screen with back-of-device gesture input by BackSwipe. We did not include other back-of-device input methods or the front-screen one-handed text entry method as base-lines, because (1) the existing back-of-device input technologies are mostly simple-gesture or tap-based, which do not support text entry, and (2) the existing one-handed text entry methods require constantly displaying a soft keyboard (e.g., [56]), or touch input on the front screen (e.g., [67]). BackSwipe was created to avoid visual and interaction interference with the front screen content, which worked work under different constraints than other front-screen one-handed text entry methods.

The testing phrases were also selected from a subset of the MacKenzie and Soukoreff phrase set [39, 57]. All the phrases in this study were different from those in Experiment 1. The participants were required to transcribe the same set of 15 phrases, divided into 5 blocks. Each block contained 3 phrases. The order of the 15 phrases was randomized. The participants were allowed to take a break after completing each block. Same as regular soft keyboards, for each back-of-device gesture input, BackSwipe generates the top candidate word on the input area and 4 additional words as suggestions. If the top candidate was not the intended word, the participant could swipe towards a direction to select the corresponding suggestion, as shown in Figure 7. If the top candidate was correct, the user tapped the back of the screen to commit it. If all the decoding results were incorrect, the participant could press the “volume down” key on the edge of the device as backspace. Before the formal test, subjects performed a warm-up session for around 5 minutes for them to get familiar with BackSwipe. The phrases in the warm-up session were different from those in the formal test.

Figure 7:

(a) A user is gesturing the word “facts” with BackSwipe. (b) When the top candidate is not the intended word, the user could swipe her finger to the direction of the word “facts” from the center of the suggestion grid to select it.

6.2. Participants and Apparatus

12 subjects (4 females, all right-handed) from 25 to 29 years old were recruited in the study. The self-reported median familiarity (1: very not familiar; 5 very familiar) with word-gesture input and Qwerty layout was 3 and 4.5, respectively. We instructed the users to completed the study using their preferred hand postures. The same ZTE AXON M device was used.

In total, the study included: 12 participants × 5 blocks × 3 phrases = 180 trials.

6.3. Results

Input Speed.

We use word per minute (WPM) to measure the input speed. This measure indicates how fast a user could input text with BackSwipe. The calculation followed the method proposed in the previous work [38]:

$$WPM=\frac{\left|S-1\right|}{T}\times \frac{1}{5},$$ (2)

where S is the length of the transcribed text in character, including spaces, and T is the elapsed time in minutes from the start of the first gesture stroke to finishing the last word in the phrase. The average input speed across all participants was 9.58 (SD=4.10) WPM. To understand how the participants performed over time, we plotted the mean (95% CI) input speed across the five blocks in Figure 8a. A repeated-measures ANOVA did not show a significant main effect of block on input speed (F_{4, 44} = 0.25, p = 0.91).

Figure 8:

Average (95% CI) of text input speed (WPM) and word error rate (%) by block.

Word Error Rate.

Gesture input on the back of the device was on word level. We measured the error rate using word error rate [5, 66]:

$$r=\frac{WD\left(E,T\right)}{\left|T\right|}\times 100\%,$$ (3)

where W D(E,T ) is the word edit distance between the entered (transcribed) phrase E and the target phrase T, and |T | is the number of words in T. The word edit distance is the minimum number of basic word-level operations needed to transform the transcribed phrase into the target phrase. The word-level operations are insertion, replacement and deletion. The average word error rate among all participants was 11.04% (SD=4.87%). Figure 8b shows the mean (95% CI) word error rate across the five blocks. A repeated-measures ANOVA did not show a significant effect of block on word error rate (F_{4, 44} = 1.70, p = 0.17).

Suggestion Usage.

We analyzed the average number of suggestion usage for each input gesture. For every gesture, if the participant used the top candidate as the output word, no (0) suggestion was used. If the participant picked one of the four suggestion words, 1 suggestion is used. The average number of suggestion used among all participants was 0.36 (SD=0.07).

Reference Keyboard Usage

We also calculated the reference keyboard usage for the text transcription task. It was calculated as the number of times the keyboard showed up on the screen over the total number of input gestures. The reference keyboard showed up an average of 0.60 (SD=0.12) times for each back-of-device gesture.

Subjective Feedback.

At the end of the experiment, participants were required to provide a numerical rating (1: least demanding, 10: most demanding) on the experiment’s mental and physical demands. The average mental demand and physical demand rating were 5.6 (SD=2.3) and 6.8 (SD=2.0), respectively. The participants also rated BackSwipe for text input on a scale of 1 to 5 (1: dislike, 5: like). The average score was 3.1 (SD=0.8).

6.4. Discussion

Our study reveals that it is feasible to input text with BackSwipe, with the average speed of 9.58 WPM. However, due to the limited flexibility, users cannot enter text as fast as on the front screen. Some users also commented that using BackSwipe over a long period of time caused fatigue. Based on the study results, it would be more practical to use BackSwipe for short or quick text entry such as replying to a short message. It is probably not appropriate for extended text entry such as writing emails. Users’ ability of using BackSwipe also differs. The input speeds ranged from 5.87 to 18.83 WPM among 12 participants. The performance varied across users.

Table 1 shows the input speed and accuracy of BackSwipe in comparison with other input methods. As shown, the performance of BackSwipe for inputting short words (e.g., command names) is comparable with other front-screen keyboard-visible input methods, while its performance decreases for inputting phrase level-text. It further suggests that BackSwipe is more suitable for short text input (e.g., command names).

Table 1:

Input speed and accuracy of BackSwipe and other input methods from literature.

Affordance	Input Method	Input Speed (WPM)	Character Error Rate(%)
Front screen keyboard visible	Gesture Keyboards [45]	25 – 31	1.0% - 3.6%
	SWiM [56]	15	0.9%
	ZoomBoard [42]	8 – 9	0.7%
	WatchWriter [18]	24	3.7%
	VelociWatch [51]	17	3.0%
BackScreen	BackSwipe (command)	18	7.5%
keyboard invisible	BackSwipe (phrases)	10	10.0%

The promising performance of BackSwipe (Table 1) and its unique affordance of being a back-of-device, gestural input method suggest that it has pros and cons compared with simple-gesture or menu based back-of-device input methods. BackSwipe is likely to be faster than other back-of-device input methods when triggering commands among a large number of candidates, because BackSwipe is essentially a recall-based command input method: a user can directly trigger a command by gesturing the command name. On the contrary, simple gestures or menu-based command input methods may involve extensive searching and navigation operations before a command can be selected (e.g., in a hierarchical menu). On the other hand, simple gesture or menu based methods might have an advantage over BackSwipe if the number of available commands is small (e.g., 2 or 3). BackSwipe increases the expressiveness of back-of-device interaction by enabling short text and word-gesture command input, while extant simple-gesture based back-of-device input methods have little support for text input. Additionally, as it is difficult to sustain long and intensive input at the back of the device, BackSwipe is recommended for short text and word-gesture command input, but not for intensive text input. Given its pros and cons, we consider BackSwipe to be a complement to the existing front-screen and back-of-device interaction methods. For example, to quickly switch to the camera application, BackSwipe allows a user to activate it by inputting the word-gesture “camera” on the back of the device, without returning to the home screen and selecting the camera icon. A user may BackSwipe a short message while she is watching a video or attending an online meeting via the smartphone. Using BackSwipe allows the user to input short text without interrupting the ongoing front-screen interaction.

7. CONCLUSIONS

We have designed and implemented BackSwipe, a back-of-device command and text input technique. It supports a user to hold a smartphone with one hand and use the index finger of the same hand to draw a word-gesture anywhere at the back of the device to enter a command and text. The key for supporting BackSwipe is the back-of-device word-gesture decoding algorithm, which infers the keyboard location from back-screen gestures, and adjusts the keyboard size to suit the gesture scales; the inferred keyboard is then fed back into the system for decoding. Based on this decoding algorithm, we have implemented BackSwipe to support command and text input. Our evaluation shows that BackSwipe is promising for supporting command input: users can enter commands at an average accuracy of 92% with a speed of 5.32 seconds/command. The text entry performance varies across users. The average speed is 9.58 WPM with some users at 18.8 WPM; the average word error rate is 11.04% with some users at 2.85%. Overall, our research contributes knowledge on decoding back-of-device gesture input and shows that BackSwipe is a feasible and promising input method for one-handed interaction on smartphones, and serves as a complement to the existing front-screen and back-of-device interaction methods.

CCS CONCEPTS.

Human-centered computing → Gestural input; Text input; Usability testing.

A COMMANDS IN EXPERIMENT 2

Thirty commands were used in Experiment 2. Each command could be triggered by one of the two words, as shown in Table 2.

Table 2:

The 30 commands and the corresponding words used to trigger them in Experiment 2.

camera, lens	copy, replicate	cut, trim	delete, remove	download, browse
edit, revise	file, folder	keyboard, typing	mail, email	print, publish
rotate, spin	search, find	network, internet	clock, timer	calculator, computer
help, assist	recent, latest	share, exchange	weather, forecast	zoom, enlarge
contact, association	date, calendar	night, dark	bluetooth, connection	photo, gallery
settings, ambience	message, letter	flashlight, torch	sound, voice	notebook, binder