Psychophysical Evaluation of Haptic Perception Under Augmentation by a Handheld Device

Abstract

Objective

This study investigated the effectiveness of force augmentation in haptic perception tasks.

Background

Considerable engineering effort has been devoted to developing force augmented reality (AR) systems to assist users in delicate procedures like microsurgery. In contrast, far less has been done to characterize the behavioral outcomes of these systems, and no research has systematically examined the impact of sensory and perceptual processes on force augmentation effectiveness.

Method

Using a handheld force magnifier as an exemplar haptic AR, we conducted three experiments to characterize its utility in the perception of force and stiffness. Experiments 1 and 2 measured, respectively, the user’s ability to detect and differentiate weak force (<0.5 N) with or without the assistance of the device and compared it to direct perception. Experiment 3 examined the perception of stiffness through the force augmentation.

Results

The user’s ability to detect and differentiate small forces was significantly improved by augmentation at both threshold and suprathreshold levels. The augmentation also enhanced stiffness perception. However, although perception of augmented forces matches that of the physical equivalent for weak forces, it falls off with increasing intensity.

Conclusion

The loss in the effectiveness reflects the nature of sensory and perceptual processing. Such perceptual limitations should be taken into consideration in the design and development of haptic AR systems to maximize utility.

Application

The findings provide useful information for building effective haptic AR systems, particularly for use in microsurgery.

INTRODUCTION

Although most augmented reality (AR) systems to date exploit visual and/or auditory perceptual channels, a need exists especially in medicine for enhanced haptic (i.e., active touch) feedback as a means of assisting users in delicate motor control. For example, in retinal microsurgery, forces imposed on the surgical instrument are generally weaker than 7.5 mN and can be felt by the surgeon only about 20% of the time (Gupta, Jensen, & Juan, 1999). Considerable engineering effort has been devoted to developing devices to augment forces at such low levels in an attempt to enhance the sensation of touch, particularly for use in microsurgery (Fleming et al., 2008; Salcudean & Yan, 1994; Stetten et al., 2011; Yao, Hayward, & Ellis, 2005). In contrast, far less has been done to characterize the behavioral outcomes of these systems: How well can the perception of small forces be augmented for the operator, and how effectively can this augmentation be put into use? In this study, using a novel handheld force magnification device (Stetten et al., 2011) as an exemplar haptic AR, we assessed the impact of augmentation on the detection and discrimination of force over a range of stimulation, and also on the perception of stiffness.

Contemporary force AR systems can be classified as “ground-mounted” or “body-mounted,” depending on the grounding configuration of their haptic interface (Biggs & Srinivasan, 2002a). Since every action causes an equal and opposite reaction, a reaction force is always needed to counterbalance the force exerted through a haptic interface. Ground-mounted haptic systems exert reaction forces by attaching the system to a massive object like a wall, floor, ceiling, or desk. For example, Taylor et al.’s microsurgical assistant system uses a tabletop robotic arm to hold the surgical tool jointly with a human operator and amplify feedback of forces measured at the tool tip (Fleming et al., 2008; Üneri et al., 2010). Grounded systems allow strong forces to be generated, but they often have disadvantages such as low portability or mechanical restrictions on manipulation in a limited and cluttered workspace. Alternatively, a body-mounted haptic device fits over the user’s own body and generates reaction forces at a location on the user other than the feedback area. Such systems can be small enough to be mounted on and move with the user’s hands (Stetten et al., 2011) or fingers (Kawasaki, Koide, Mouri, & Endo, 2010). One exemplar, the handheld force magnifier developed by Stetten et al. (2011), is a portable exoskeleton-like system that grounds the amplified fingertip forces on the back of the user’s own hand. In comparison to ground-mounted systems, body-mounted devices afford high manipulation capacities at a wide variety of locations and orientations.

In comparison to the volume of research aimed at developing force AR systems, far less has been done to characterize their behavioral outcomes. Validation studies have demonstrated that such AR systems can help users reduce hand tremor and better perform delicate procedures like membranectomy and cannulation of retinal vessels with improved success rates and higher repeatability (Balicki et al., 2010; Fleming et al., 2008; Üneri et al., 2010). Whereas there is no question that such improvement is mediated through the enhanced signals to the sensorimotor system, little research has looked directly at the impact of force augmentation on perceptual functions. Ideally, such augmentation should amplify force but impose no perceptual distortions within the range of operation. But in reality, no AR system can function perfectly because of inertia, friction, hysteresis, delay, and other nonlinearities in haptic interfaces and control systems. Research is thus needed to quantify the user’s experience of such augmentation, to examine whether it is unequivocally positive as designed, and more important, to identify the cause of perceptual distortions if found. Next we discuss perceptual limitations and draw potential implications for the effects of force augmentation.

An understanding of the effects of force AR is particularly important in the context of perceptual limitations at threshold (barely perceptible) and suprathreshold (clearly perceptible) levels. A beneficial outcome that is generally intended from magnification is to bring otherwise imperceptible stimulation into conscious experience, that is, to cross the “absolute threshold.” However, what constitutes the threshold for detecting force is a rather complex issue. Numerous psychophysical experiments have attempted to measure the threshold, with varying results, as shown by outcomes from representative experiments in Table 1 (we summarize the work most relevant to this study; for more extensive review, see Jones, 2000, and Jones & Lederman, 2006). The threshold varies considerably over the body, with the highest sensitivity being found on the forehead and lowest in the lower extremities (Weinstein, 1968). Even at a particular body site, the threshold can be significantly affected by the characteristics of the force stimulation and how it was presented, and by individual differences such as gender. Examples of factors that can affect the threshold measurement include the frequency of the force stimulus (Israr, Choi, & Tan, 2006) and the rate of force application (Schoo, van Steenberghe, & de Vries, 1983). We suggest that although magnifying near-threshold forces should generally facilitate detection, the variations in threshold make it important to verify that magnification is effective at threshold levels within the context of intended use. Experiment 1 provides this test with our exemplar device, the handheld force magnifier developed by Stetten et al. (2011).

Table 1.

Absolute and Differential Thresholds of Force Perception

Measure	Publication	Procedure	Value
Absolute threshold	Weinstein, 1968	Using von Frey hairs to apply normal forces to fingertips	0.54 mN (male) 0.19 mN (female)
	Dosher & Hannaford, 2005	Tangential force pulses (50–150 ms) applied to fingertip regions of 3–5 mm diameter	22.5–28.9 mN
	Mesa-Múnera et al., 2012	Detecting force while drawing circles with Phantom device	247.8 mN
Differential threshold (just noticeable difference)	Pang, Tan, & Durlach, 1991	Finger pinching	5%–10%
	Jones, 1989	Force matching with elbow flexors	5%–9%

Experiment 2 examines the impact of augmentation on the perception of suprathreshold forces. A measure for sensitivity to suprathreshold forces is the discrimination threshold, as quantified by the just noticeable difference (JND) in the percentage of a reference force. The JND has been found to be quite high, 15% to 27%, for forces below 0.5 N but reduces to 5% to 10% for forces larger than 0.5 N (Jones, 2000). Beneficial outcomes then may be expected for judging weak forces via augmentation if magnification brings them above 0.5 N, because these forces can be better discriminated and distinguished via the augmented output. On the other hand, some studies point to possible disadvantageous effects. Research has shown that the sensation of force grows as a power function of its intensity (Stevens, 1975), and the value of the exponent varies from 0.7 to 2.0 due to differences in the range of stimuli, experimental procedures, and the kinetic aspects of force generation (Jones, 1986). For example, Curtis, Attneave, and Harrington (1968) reported an exponent of 0.746 for estimating a set of eight weights between 10 and 310 g (i.e., 0.10 N to 3.04 N). Accordingly, an opposite prediction could be made for the perception of weak forces in this range: If the force sensation becomes more compressive with increasing force intensity, one may expect that a linear augmentation could make force discrimination more difficult. We examine the effects of magnification on force discrimination in Experiment 2.

Perceived force also provides critical input to perception of object properties like stiffness. Stiffness is considered as a higher-order perceptual variable that is not directly perceived but estimated as a ratio of perceived force and deformation (Klatzky & Wu, 2014). To the extent that magnifying forces is beneficial to force perception, one would expect stiffness judgments to be similarly aided. If so, force AR then will have great potential in clinical practice for assisting the physician, not only in performing surgical manipulations, but also in diagnosis as with stiffness-based detection of tumors. Limits in stiffness perception, for example, are indicated by the low detection rate of 39% to 59% for breast cancer examination by palpation (Shen & Zelen, 2001).

However, force magnification may not have consistently positive effects across a broad range of stiffness levels, due to the mechanisms underlying perception. When the stiffness of a surface is determined by active fingertip pressure, in particular, cues arise from a combination of cutaneous (skin) and kinesthetic (muscle/tendon/joint) receptor responses. Cutaneous deformation patterns on the fingertip are informative for small forces that do not cause the fingertip to “bottom out” during exploration (Srinivasan & LaMotte, 1995). As the force increases, cutaneous information gets saturated and stiffness perception will rely more on relatively weaker kinesthetic cues. Srinivasan and LaMotte (1995) reported a lower bound on the softness JND of 12% when cutaneous and kinesthetic information was present, but their participants could not discriminate much larger compliance differences on the basis of kinesthesis alone. To the extent that force magnification causes the perception of stiffness to rely more on kinesthesis and less on cutaneous cues, it could have detrimental effects. Experiment 3 assesses the effects of magnification on stiffness perception.

In this study, we focused on augmenting the perception of weak forces (<0.5 N). The selection of this threshold was based on previous findings (see Jones, 2000, for review) that the JND substantially increases for forces less than 0.5 N, as occur during medical microsurgery. As noted earlier, a first-generation handheld force magnification device (Stetten et al., 2011) was used as an exemplar device, and three experiments were conducted to assess the effectiveness of augmentation using the behavioral measures we have reviewed. The handheld magnifier senses the pushing force applied to the tip of its handle, and electronically amplifies the signal and then generates a proportionally magnified output on the handle by means of an electromagnetic actuator (see the method section of Experiment 1 for more technical details). The impact of magnification on the absolute and differential thresholds of force perception was examined in the first two experiments, respectively. The third experiment examined the perception of stiffness through magnified force. Assuming that the effects of magnification would pass throughout the sensory systems to the final perception without loss or distortion, the user’s subjective experiences should match well with the physical output of the device. By comparing the behavioral outcome to the physical augmentation, we were then able to assess the loss in the augmentation’s effectiveness. In the general discussion section, we discuss some issues that affected the transmission of magnified force information with this device and draw inferences for force AR more generally.

EXPERIMENT 1: EFFECTS OF AUGMENTATION ON FORCE DETECTION

In this experiment, the participants’ ability to detect force (i.e., the absolute threshold) was measured with or without the use of the force magnifier and compared to that obtained in unaided perception. By comparing these thresholds, we could estimate the behavioral gain and compare it to the physical magnification. As explained later, the magnifier has a physical amplification of 3.4×. We thus expected that the observed threshold for detection should be reduced correspondingly by a factor of 3.4, if the magnification could be effectively transmitted to perception.

Method

Participants

Eight men and four women, aged 22 to 36 years old, participated in this experiment with informed consent. To eliminate the possible influence of handedness, all participants were right-handed by self-report. They were naive to the purpose of the study.

Experimental apparatus

Figure 1 shows the 1-DOF force magnifier used in this study. The device includes a handle to be held by the user’s fingers, a force sensor and an actuator placed at opposite ends of the handle, a brace for mounting the device on the user’s hand, and a control system. The handle is composed of the body of a 1 ml syringe attached to a piece of 0.25-inch brass tube. At the tip of the syringe, a commercial force sensor (Honeywell FS01, 0–6.7 N) is mounted to measure the force between the handle and a target. At the other end of the handle, a stack of 8 permanent rare-earth magnets (3/16 inch Radio Shack 64–1895) are placed inside the brass tube and inserted into a custom coil (250 ft. of 30 gauge wire, 25 ohms, approx. 2,360 turns). The magnets and coil act as a voice coil actuator to actuate the handle: The coil is powered by an electrical current from the control system, inducing a Lorentz force on the magnets and hence on the handle. The actuator is attached by a dual gimbal to a brace, which is mounted to the back of a wrist splint strapped to the user’s right hand. The dual gimbal increases the flexibility of use by permitting two degrees of freedom of rotation (forward/backward tilting and left/right rotating relative to the hand) while maintaining a rigid reference against which to generate force in the range direction. The wearable part of the device weighs 163.6 g. In addition, a separate control system is used to control the electrical current in the coil actuator and generate a magnified output of force with an adjustable gain ranging from 1.0× to 5.8×. For more technical details, please see Stetten et al. (2011).

Figure 1.

The force magnification device used in this study. The device uses a sensor to measure the force (f) between the tip and the target, which is amplified to produce f’ = k*f (k was set to 2.4 in this study) in the same direction on the handle using an actuator mounted on the back of the hand via a brace. The total force applied to the user’s hand was thus (F_total = f’ + f), producing a magnification of (1 + k) or 3.4 here.

When using the magnifier, the user holds the handle as shown in Figure 1 and feels a sum of two forces through it: the input force (f) and a Lorentz force exerted on the magnets (f’). The Lorentz force (f’), generated by an electrical current proportional to the force signal, is in the same direction as the input force: f’ = kf. Thus the overall force applied to the handle and the user’s hand is then (f’ + f), producing a magnification power of (1 + k). In this study, the proportionality factor (k) was set to 2.4. Therefore the physical magnification produced by the device was 3.4.

Experimental stimuli

The stimuli were upward (countergravitational) forces in this experiment. All stimuli were generated using a Magnetic Levitation Haptic Device (MLHD, Model# Maglev-100, Butterfly Haptics LLC. Pittsburgh, PA, http://www.butterflyhaptics.com). The MLHD uses the Lorentz forces produced by six current-carrying coils to levitate a handle in a magnetic field and generate high fidelity force output (Berkelman & Hollis, 2000; Hollis & Salcudean, 1993). Since it has no motors, gears, bearings, or linkages present, the MLHD is free of static friction and able to render forces with a precision of 0.02 N. The MLHD was connected to a control computer through a 100 Mbps Ethernet cable, through which commands were sent to generate the desired stimuli and the participant’s keypad responses were recorded.

Experimental design and procedure

The participants’ ability to sense the presence of a force was tested in three conditions: magnifier-ON, magnifier-OFF, and control. In the magnifier-ON and magnifier-OFF conditions, the participant put the device on his or her right hand and held the syringe part between the thumb and index finger, as shown in Figure 2a. He or she was also instructed to fully extend the force magnifier so that the magnet position within the coil did not vary from trial to trial, guaranteeing predictable forces generated by the actuator. In the control condition (Figure 2b), the participant held an unmodified syringe in the same way that the magnifier was held in the experimental conditions and felt the force transmitted through it.

Figure 2.

Illustration of the experimental setup. The user’s perception of force was compared across the magnifier-ON/OFF conditions (a) and control condition (b). A magnetic levitation haptic device (MLHD) was used to produce the experimental stimuli.

The participants were tested individually. At the beginning of the experiment, they were asked to adjust the height of the chair to allow their right forearm to rest on the MLHD’s rim in a comfortable position, aided by a memory foam cushion. They held the force magnifier or the syringe still with the right hand, kept it in contact with the MLHD handle, and felt a stimulus force through it. The method of limits was used to measure the force-detection threshold. The initial force was well above (0.3–0.4 N) or below (0.0 N) the threshold in the descending or ascending series, respectively. The participant adjusted the force by pressing the “−” or “+” button with the left hand until the force seemed to just disappear or appear: At that point, he or she felt as if somebody who was trying to pull the syringe or the magnifier out of his or her fingers with a gentle upward force had stopped in a descending series, or somebody just started to pull the syringe or the magnifier in an ascending series. Upon each key press, the force stimulus was adjusted by 0.02 N with a probability of 70% to prevent the participants performing the task by counting the number of key presses. The interval between key press and force adjustment was randomized between 100 ms and 150 ms. The participants were tested with eyes closed and ears covered by noise-cancelling headphones to prevent the possible use of non-haptic cues. Under each experimental condition, the descending and ascending limits, respectively, were measured twice and the threshold was then calculated as the average of the four limits. The participants were tested first in the control condition and then the magnifier-ON and magnifier-OFF conditions with counterbalanced order. The experiment took about half an hour.

Results

Figure 3 plots the mean detection thresholds obtained in the three conditions. The effect of force magnification was evident. When the magnifier was on, the mean threshold was reduced to 0.07 N, much lower than that observed in the magnifier-OFF (0.20 N) or control (0.16 N) conditions. A one-way, repeated measures ANOVA confirmed that such differences were significant, F(2, 22) = 22.92, p < .001, partial η² = .68. Pairwise comparisons with Bonferroni correction revealed that the magnifier-ON threshold was significantly lower than the magnifier-OFF, t(11) = 5.66, Bonferroni-adjusted p < .001, d = 1.68, or control threshold, t(11) = 4.42, Bonferroni-adjusted p = .003, d = 1.29.

Figure 3.

Measured force detection thresholds in the control, magnifier-ON, and magnifier-OFF conditions. Error bars represent the standard error of the mean.

To quantify the effectiveness of magnification, a behavioral gain was calculated as the ratio of the thresholds obtained in the magnifier-ON and magnifier-OFF conditions. The average gain across all participants was 2.9, which did not significantly differ from 3.4, the actual magnification power of the device, one-sample t test: t(11) = 1.19, p = .26, d = 0.35. This suggests that the magnification induced by the device is well perceived by users for the stimulation at this level, although the mean behavioral gain is a bit smaller than the physical magnification.

Next, a comparison was made between the magnifier-OFF and control data to examine if wearing the device could cause some detrimental effects. Although the threshold was slightly higher when the magnifier was worn but turned off (0.20 N vs. 0.16 N), the difference was statistically insignificant, t(11) = 2.49, Bonferroni-adjusted p = .09, d = 0.72. Note also that the thresholds were much higher than the values found in the experiments where force was applied directly to the fingertip (Weinstein, 1968: < 1 mN for detecting force applied normally using von Frey hairs; Dosher & Hannaford, 2005: < 30 mN for detecting of tangential force applied to fingertip regions), but they matched well with the value (247.8 mN) reported by Mesa-Múnera, Ramirez-Salazar, Boulanger, Bischof, and Branch (2012), where the force was felt by the participants holding the pen of a PHANTOM haptic device with multiple fingers. Clearly, the detection performance is significantly influenced by factors like size of contact area, direction of force stimuli, use of tools, and others. In principle, such factors can affect all force AR devices, which will be further discussed in the general discussion section.

EXPERIMENT 2: EFFECTS OF AUGMENTATION ON FORCE DISCRIMINATION

Whereas Experiment 1 measured the minimum detectable value, Experiment 2 examined the effectiveness of magnification in assisting with force discrimination at suprathreshold levels. Here the participants’ force JND was measured with or without the force magnifier and compared to that obtained in the control conditions to determine the effects of perceptual augmentation. Research has shown that people’s ability to distinguish between two similar forces improves with intensity within certain limits, and the force JND tends to be greater for near-threshold stimuli than for larger stimuli. Therefore, as magnification increases the force level, we expect that it can help the user better sense the differences among the small input forces via the magnified output produced by the device.

Method

Participants

Sixteen naïve participants (10 men and 6 women, aged between 18 and 38 years) were tested with informed consent. All were right-handed.

Experimental apparatus

The apparatus was the same as the previous experiment.

Experimental design and procedure

Participants were tested individually and their ability to discriminate forces was assessed in the following four conditions. JNDs were measured for a reference force of 0.3 N in the magnifier-ON, magnifier-OFF, and control conditions. In addition, a fourth magnifier-OFF condition was tested with a reference force of 1.02 N, equivalent to the output produced in the magnifier-ON condition for a force of 0.30 N at the magnification of 3.4×. The test order of these conditions was counterbalanced across participants using a Latin square design.

JND was measured by using an unforced three-up/one-down adaptive procedure (Kaernbach, 2001) that targeted 75% correct detections. On each trial, a pair of force stimuli (reference and test) was presented to the participant, one at a time, along with a color label in the form of a yellow or blue square shown on a LCD screen. The forces (reference or test) and color labels (yellow or blue) were assigned randomly to the first and second stimulus. The participant could switch between two stimuli as many times as desired by first removing the syringe or magnifier from the MLHD handle and then pressing a button labeled “Switch.” A transition phase with a random duration of 0.4 to 1.0 s was inserted between switches, during which one stimulus (i.e., the force and its color label) gradually changed to another. The participant’s task was to judge which force felt stronger and then press a colored button corresponding to that stronger stimulus or a third button labeled “Unsure” if he or she could not tell the difference. The participant’s response was recorded and used to adjust the test force on the next trial using the following algorithm: the difference between the reference and test forces would be decreased by one step if the participant made a correct judgment, increased by three steps if erroneous, or increased by one step for an “unsure” response. A reversal was tagged when a response produced a change in the direction of adjustment (i.e., an erroneous or unsure response preceded or followed a correct response). A JND session was terminated after eight reversals, and a threshold was estimated by averaging the test values between the fourth and eighth reversal. In each experimental condition, the threshold was measured twice in both ascending and descending directions, and the mean of two measurements was calculated as the JND. The ascending and descending sessions were interleaved to preclude any predictive effects.

In this experiment, the initial difference between the reference and test forces was 24%. The initial adjustment step was 8% and halved at the second and fourth reversals. Typically, the participants finished one trial in less than 20 s and a whole JND session in less than 15 min. There were 5-min breaks between the experimental sessions. The entire experiment took approximately 1 hr.

Results

JNDs were compared using a one-way repeated-measures ANOVA, followed up with Bonferroni pairwise comparisons. The ANOVA revealed a significant difference in mean JNDs across the experimental conditions, F(3, 33) = 11.59, p < .001, partial η² = .44. As shown in Figure 4, the mean JND for a reference force of 0.30 N was much lower in the magnifier-ON condition (11.04%) as compared to the magnifier-OFF (15.39%) and control (15.15%) conditions. Pairwise comparisons confirmed that such differences were statistically significant, magnifier-ON versus magnifier-OFF: t(15) = 3.10, Bonferroni-adjusted p = .04, d = 0.77; magnifier-ON versus control: t(15) = 3.76, Bonferroni-adjusted p = .01, d = 0.94. The observed JND was similar in the control and magnifier-OFF conditions, and no significant difference was detected, t(15) = 0.16, Bonferroni-adjusted p > .99, d = 0.04.

Figure 4.

Mean JND of force discrimination as a function of the experimental conditions. Error bars represent ±1 standard error.

Note that all forces were magnified by 3.4 times in the magnifier-ON condition and the reference force of 0.30 N was augmented to 1.02 N. Thus we tested the participants in a second magnifier-OFF condition with a reference force of 1.02 N. The JND obtained in this magnifier-OFF condition was 9.58%, which was much lower than that obtained in the same condition but with a lower reference of 0.30 N, 9.58% versus 15.39%, t(15) = 4.91, Bonferroni-adjusted p = .001, d = 1.23. This agrees with previous research showing increased JNDs for forces weaker than 0.5 N (Jones, 2000). Of importance, when comparing this magnifier-OFF 1.02 N reference JND to the magnifier-ON JND obtained with a reference force of 0.30 N, no significant difference was found, 9.58% versus 11.04%, t(15) = 1.83, Bonferroni-adjusted p = .53, d = 0.46. These results clearly show that the force magnifier increases the sensation of small forces by amplifying the weak input, bringing it to a more perceptually discriminable level, and thus enhancing the user’s ability to detect small differences in the input.

EXPERIMENT 3: EFFECTS OF AUGMENTATION ON STIFFNESS ESTIMATION

The two experiments so far have shown that force magnification can improve the user’s ability to detect and tell apart small forces. Next we consider how force magnification can help the user in the perception of mechanical properties such as stiffness. By definition, stiffness is the ratio of the force to the deformation caused by this force, and is judged from the perceived force and deformation. Thus we expect that stiffness judgments would be similarly aided by magnification as in the perception of force, although there might be a loss in effectiveness given the additional factor of deformation. Here we used the method of magnitude estimation to quantify stiffness perception and characterize the effects of force magnification. We chose this task because the capacity of humans to scale the magnitude of stiffness has received little attention and also because it directly reflects the impact of force magnification. Of importance, Varadharajan, Klatzky, Unger, Swendsen, and Hollis (2008) found no significant contribution of vision to the perception of stiffness magnitude (compare significant visual influence in stiffness discrimination). Thus, the task is very suitable for characterizing the performance of haptic systems. As in Experiment 1, we estimated the behavioral gains from the magnifier-ON and magnifier-OFF data and compared them to the physical magnification to evaluate how effectively the magnified stiffness was transmitted to perception.

Method

Participants

The twelve participants were the same as those in Experiment 1.

Experimental apparatus

The apparatus was the same as the previous experiments.

Experimental design and procedure

This experiment had two independent variables: device conditions (the magnifier-ON, magnifier-OFF, and control conditions as in Experiments 1 and 2) and stiffness of virtual springs (20, 40, 60, and 80 N/m). The method of magnitude estimation was used to quantify stiffness perception. Thus the dependent variable is subjective rating of stiffness. The participants were instructed to freely interact with a virtual spring on each trial and then give a numerical estimate of the perceived stiffness. They were also clearly instructed that any number could be used to report their impression of stiffness and the stronger the felt stiffness, the larger the number should be. In addition to the four stiffness values tested across all conditions, two springs of 100 and 200 N/m were tested only in the control condition to reduce range effects resulting from the magnification. All springs could be compressed by up to 6 mm, resulting in reaction forces up to 0.12 to 0.48 N for the stimulus springs. The compression range was enforced by showing a red dot on a LCD screen to indicate the handle position of the haptic interface and instructing participants to avoid overcompressing the virtual spring and moving the dot out of the screen. (Although the compression limit was set to 6 mm, the virtual springs could be actually compressed by as much as 12 mm.) Each stimulus was tested three times in each condition, and the repetitions occurred in different testing blocks in random order. The magnifier-ON and magnifier-OFF trials were intermixed so that the participants had no knowledge of the status of the device. The order of control and magnifier conditions was counterbalanced across participants. The experiment lasted about 1 hr.

Results

The data were normalized by dividing each judgment by the participant’s mean within a given condition, and then multiplying by the grand mean for all participants (Stevens, 1975). This produces a measure independent of the participant’s arbitrary choice of scale. Figure 5a shows the mean estimated magnitude, averaged across all participants, for the stiffness tested in the three conditions. Although the magnifier-OFF and control curves largely overlapped, the estimates obtained in the magnifier-ON condition were significantly higher. In addition, the magnifier-ON curve appeared more compressive than the other two curves, indicating a gradual reduction in the perceived augmentation with the stimulus intensity.

Figure 5.

(a) Mean estimates of perceived stiffness. Error bars represent ±1 standard error. (b). Replot of the magnifier-ON and control data, including the stiffness tested only in dummy trials in the control condition. The magnifier-ON data were replotted versus the augmented stiffness produced by the force magnifier (i.e., 3.4× stiffer as predicted from the augmented force).

Behavioral gains were estimated for each virtual spring by relating each participant’s magnifier-ON judgments to his or her magnifier-OFF judgments. The gain was found to be largest for the softest spring and lowest for the stiffest one. The average gain for the spring of 20 N/m was 2.7, which was not statistically different from the physical magnification of 3.4, one-sample t test: t(11) = 1.08, p = .30, d = 0.31. For the other springs, the mean gain reduced from 2.3 to 2.0, and to 1.7. All these gains were significantly lower than the physical magnification, one-sample t tests: t(11) = 3.79, p = .003, d = 1.09 for the spring of 40 N/m; t(11) = 4.53, p = .001, d = 1.31 for the spring of 60 N/m; t(11) = 15.19, p < .001, d = 4.38 for the spring of 80 N/m. Thus the magnifier-ON curve, if horizontally rescaled by multiplying the stiffness values by 3.4, eventually diverged from and fell below the curve obtained in the control condition, as illustrated in Figure 5b.

As in the previous experiments, we focused the statistical analysis on the following two comparisons: the comparison between the magnifier-ON and magnifier-OFF conditions to validate the beneficial effects of force augmentation and the comparison between the control and magnifier-OFF conditions to examine if there were any detrimental effects caused by the wearable device. Two-way (Device × Stiffness) repeated-measure ANOVAs were performed on the raw data (i.e., not normalized as described earlier), followed by Bonferroni pairwise comparisons for significant main effects. The comparison of the magnifier-ON versus magnifier-OFF data found significant main effects of Device, F(1, 11) = 12.15, p = .005, partial η² = .53, and Stiffness, F(3, 33) = 14.56, p < .001, partial η² = .57, but no significant interaction between them, F(3, 33) = 2.47, p = .08, partial η² = .18. Pairwise comparisons further revealed that the effects of force augmentation reached significance at all levels of Stiffness, t(11) > 3.11, Bonferroni-adjusted p < .04, d > 0.89. We also analyzed the data using a 3 (Device) × 4 (Stiffness) repeated-measures ANOVA, followed by Bonferroni pairwise comparisons and tests of the simple main effects of Device. The analyses found the same pattern of results. In contrast, the magnifier-OFF versus control comparison found that the main effect of Stiffness, F(3, 33) = 18.39, p < .001, partial η² = .63, was significant, but neither the main effect of Device, F(1, 11) = 0.10, p = .76, partial η² = .01, nor its interaction with Stiffness, F(3, 33) = 2.64, p = .07, partial η² = .19, reached significance. This suggested that the wearable device, when turned off, produced little impact on the perception of stiffness.

Although the results have demonstrated the effects of force augmentation in stiffness perception, the question remains, how effectively is the augmented force perception used in stiffness estimates? Note that stiffness is judged on the basis of perceived force and deformation. Therefore the augmentation effects might be lowered in stiffness perception due to additional variability in deformation perception. To examine this, we also tested the participants in a pilot force-estimation experiment under the same experimental conditions, in which the force stimuli ranged from 0.1 N to 0.4 N, roughly matching the maximum resisting forces (0.12–0.48 N) produced by the virtual springs in the current experiment (results were presented in preliminary form in Stetten et al., 2011). Thus we were able to calculate the individual’s behavioral gains in that force estimation task and make a direct comparison of his or her performance in the two tasks to determine if the processing in stiffness perception caused further loss in the effectiveness of the magnified forces. The behavioral gains ranged, respectively, from 2.6 to 1.9 for force estimation and from 2.7 to 1.7 for stiffness estimation. The same trends were shown in both tasks: The gain was found to be largest for the weakest stimulus but smallest for the strongest stimulus. Four paired t tests were then conducted, and none found any statistically significant difference between the two sets of gains, t(11) < 1.13, p > .28, d < 0.33, although the mean behavioral gain was generally a bit smaller in estimating springs than forces. That is, force magnification effectively changes the perception of stiffness, as would be predicted from its effects on the perception of force. One may also expect that such magnification effects can be similarly conveyed to action control or judgments of other perceptual entities like viscosity that relies on the perceived force.

GENERAL DISCUSSION

Our results clearly demonstrate that the perceptual augmentation produced by the force magnification device is well perceived by users in terms of their ability to detect and discriminate small forces or use the perceived force in stiffness perception. In particular, the magnification was found to work best for small forces. The observed behavioral gain was 2.9 in Experiment 1 for detecting forces at the threshold level and 2.7 in Experiment 3 for estimating the softest stiffness. These gains did not differ significantly from the magnification power of 3.4.

The results also reveal a trend that the effectiveness of magnification gradually declines with the stimulus intensity. Take Experiment 3 as an example. As the stiffness and hence the interaction force increased, the observed behavioral gain reduced from 2.7 to 1.7. Although the device used in this study is far from perfect and has many limitations, we do not think that such loss in gain can be simply attributed to the device. We have carefully calibrated the device so that it works quite linearly within the operating range (see Figure 5 of Stetten et al., 2011, for illustration). Static friction exists between the handle and coil of the actuator. (The magnifier was held and used by the operator fully extending its handle to prevent motion and kinetic friction.) This may partially explain why the observed gains were smaller than the physical magnification. But the effects of friction would be more significant for weak than for strong force. That is, it would cause more loss in gain for weaker stimuli, contradictory to the trend observed here.

The loss in the behavioral gain may reflect a compressive growth of force sensation within the range tested. Curtis et al. (1968) used a set of eight weights of 10 to 310 g (0.10–3.04 N) as the stimuli and asked their subjects to estimate the magnitude of weights and the magnitude of differences between weight pairs. They found that the estimates obtained from both tasks were best modeled by power functions with exponents of 0.746 and 0.645, respectively. In our Experiment 3, the maximum resisting forces produced by the virtual springs was 0.48 N at a compression of 6 mm or 1.63 N after a magnification of 3.4×. This fell into the stimulus range tested by Curtis et al. (1968). Consistent with their findings, the magnifier-ON and magnifier-OFF curves in Figure 5 were found to be fitted very well by two power functions with exponents of 0.60 (r² = .993) and 0.81 (r² = .995), respectively. Such perceptual distortion would also be expected in other force AR systems that linearly amplify weak forces. As the input increases, a linear force magnifier yields a proportional increase in the physical output, but a slower (exponent < 1) growth in the perceptual experience. This discrepancy between the physical and perceptual outcomes increases with the increasing input force, thereby causing an increasing reduction of the behavioral gain.

We further postulate that the loss in the behavioral gain reflects the nature of sensory and perceptual processing that is generally relevant to the potential effectiveness of force AR devices. First, the loss may be partly accounted for by a shift in utilization of cues from cutaneous mechanoreceptors to kinesthetic receptors. As explained in the introduction, small forces are sensed mostly through the deformation of the skin. The perception of large forces relies more on the information from receptors in muscles, tendons, and joints that have relatively lower sensitivity. As force is amplified, the perceptual system then tends to rely more on kinesthetic than cutaneous sensations, leading to reduced performance. Second, interference may occur at the receptor level between the stimulus and other interaction forces. As illustrated in Figures 1 and 2, the operation of the force magnifier or other handheld devices requires the user to hold the device with grip force that is normal to the skin surface, while feeling the augmented stimulus through tangential skin stretch. The SAI and SAII receptors respond differentially to normal and tangential skin deformation, but research has shown that both grip and tangential forces can activate most of the two types of receptors, producing a masking effect on the perception of tangential force (Birznieks, Jenmalm, Goodwin, & Johansson, 2001). Such an effect increases with the forces applied and can lead to a loss in sensitivity. For example, Wheat, Salo, and Goodwin (2004) reported that the JND of tangential force discrimination increased from 16.3% to 25.2% when the reference increased from 1.0 N to 1.6 N along with a proportional increase of the accompanying normal force from 2.5 N to 4.0 N.

These results suggest some important factors that should be considered in the design and implementation of force AR systems, particularly for those handheld devices. For example, after appropriate psychophysical calibration, it might be possible to compensate for the loss in the behavioral gain so as to produce the subjective experience of a constant magnification over the stimulus range. Of importance, to maximize the effectiveness of force magnification and enhance user performance, it is critical to have force effectively transmitted from the device to the user. Factors like the mechanical impedance of skin surface and the device’s geometric and material attributes then should be considered. Taking again our device as an example, its effectiveness can be further improved in a number of specific ways. As the device is pinch-held by the user, the force produced by it is transmitted and felt as tangential stimulations. At the fingertips, however, the impedance to tangential stretch is about fivefold higher than that for normal deformation (Diller, Schloerb, & Srinivasan, 2001), suggesting that such tangential stimulation is not the optimal way to transmit information. In contrast, impedance at the forearm is significantly smaller in the tangential direction than in the normal direction (Biggs & Srinivasan, 2002b). It might be possible to design the device to be mounted on the forearm and display the tangential reaction force with higher effectiveness. In addition, note that the handle of the device is the body of a syringe, which has a smooth surface with very fine texture. This also influences the perception of axial forces transmitted along the handle: The smoother the surface, the greater the grip force required and the more difficult it is to perceive the lateral shear force due to the masking effects of the grip force (Flanagan & Wing, 1997). These factors likely contribute to the reduced perception of forces transmitted along the handle. With these considerations in mind, we will improve the design in the next generation of our force magnification device to maximize the effectiveness of perceptual augmentation.

In addition to these technical considerations, certain behavioral strategies can be applied to improve the effectiveness. Lederman and Klatzky (1987) have identified a set of motor patterns, called exploratory procedures, which are specialized for effectively extracting information about certain object properties. In this study, we also noticed the use of such procedures with the force magnification device. For example, when estimating the spring stiffness, we found that the most convincing sensation of magnified stiffness was achieved by moving the entire device with the upper arm and shoulder, instead of the hand or only fingers. By moving the device with the shoulder, one can maximize the moment arm and the torque created on the shoulder joint, and use the torque as an additional cue to the force and stiffness (Wu, Klatzky, & Hollis, 2011).

Another question that might be raised about the utility of a handheld device is whether users might adapt to the forces they encounter, either from the weight of the device or from its amplification effects. Force perception is clearly labile. For example, constant perceived effort in handgrip is associated with declining physical force over even a 45-s period (Cain, 1973). Longer periods of lifting forces by hand lead to increases in perceived weight and effort as fatigue develops (Burgess & Jones, 1997). Weight perception is also known to be highly susceptible to the comparison context, as shown by Helson’s classic adaptation-level theory (e.g., Helson, 1947). Feedback effects are also powerful. In experiments by Brewer, Klatzky, and Matsuoka (2006), visual feedback regarding force magnitude produced by the index finger was gradually distorted in sub-JND steps, so that more or less force was required to produce a given feedback level on the screen. When instructed to produce a target force, participants followed the feedback without awareness of the distortion, despite its cumulative effects well exceeding a JND, and the influence of distorted feedback persisted on trials where the visual display was withdrawn. Given the complexity of these effects, although adaptation may affect the effectiveness of force augmentation, it is difficult to predict its time course or even the direction of the effect. Perceived force might increase due to fatigue, for example, or it might decrease due to resetting the comparison level as higher forces are experienced. The weakness of forces encountered here (and in the intended application domain), as well as the finding that the impact of the device’s weight on user performance was not significant, suggest that fatigue effects at least would be minimal. In any case, future research is clearly needed to evaluate the effects of force augmentation over extended use.

In summary, the experiments reported here assessed the behavioral impact of force magnification and showed that it could effectively improve the operator’s ability to sense and discriminate between small forces and to judge the stiffness of targets. We also observed the influence of the haptic sensory and perceptual systems on the augmentation’s effectiveness over the stimulus range. These findings provide useful information for building effective force AR systems, particularly those body-mounted or handheld devices for use in microsurgery, a realm in which many very delicate operations are performed, often devoid of a useful sense of touch.

Key Points.

• Force augmentation is effectively transferred to haptic perceptual judgments, not only of force but also of other perceptual attributes such as stiffness.

• The perceptual effectiveness of force augmentation gradually declines with stimulus intensity in comparison to a control.

• Factors such as skin impedance and the device’s geometric and material attributes should be considered in the design of force AR systems to maximize user experience.

Biographies

Bing Wu received a PhD in experimental psychology from University of Louisville. He is an assistant professor in Arizona State University’s College of Technology & Innovation. His research interests are in spatial perception/cognition, the visual and haptic control of action, and human factors issues in real, teleoperated, or simulated medical operations.

Roberta Klatzky received a PhD in psychology from Stanford University. She is a professor of psychology and human computer interaction at Carnegie Mellon University. Her research interests are in human perception and cognition, with special emphasis on spatial cognition and haptic perception.

Randy Lee received an MS in biomedical engineering from Carnegie Mellon University. He is a doctoral candidate in the Department of Bioengineering at the University of Pittsburgh. His research interests include medical device design, surgical engineering, and psychophysics.

Vikas Shivaprabhu received an MS in electrical engineering from the University of Pittsburgh. He is a doctoral candidate in the Department of Bioengineering at the University of Pittsburgh. His research interests include medical device design and biomedical image analysis.

John Galeotti received a PhD in robotics from Carnegie Mellon University. He is a systems scientist in robotics and an adjunct assistant professor in biomedical engineering at Carnegie Mellon University and bioengineering at the University of Pittsburgh. His research interests are in both biomedical optics and biomedical image analysis and visualization, especially for intraoperative guidance.

Mel Siegel received a PhD in physics from the University of Colorado Boulder. He is a faculty member in robotics and an affiliated faculty member in human computer interaction at Carnegie Mellon University. His research interests are in sensing, sensors, perception, and display systems in robotics contexts.

Joel S. Schuman received an MD from the Mount Sinai School of Medicine. He is professor and chairman of ophthalmology at the University of Pittsburgh School of Medicine and the director of UPMC Eye Center. His research interests include imaging of the eye, laser-tissue interactions, aqueous outflow, and clinical pharmacology.

Ralph Hollis received a PhD in physics from the University of Colorado. He is research professor of robotics and electrical and computer engineering at Carnegie Mellon University, and founding director of the Microdynamic Systems Laboratory at Carnegie Mellon University, where his research centers on haptics, agile manufacturing, and dynamically stable mobile robots.

George Stetten received an MD from SUNY Syracuse and a PhD in biomedical engineering from University of North Carolina, Chapel Hill. He is professor of bioengineering at the University of Pittsburgh and research professor at the CMU Robotics Institute. He is also the director of the Visualization and Image Analysis Laboratory and the Music Engineering Laboratory. His research is in image guided intervention, haptics, and image analysis.