Measuring tactile cues at the fingerpad for object compliances harder and softer than the skin

Abstract

Distinguishing an object’s compliance, into percepts of “softness” and “hardness,” is crucial to our ability to grasp and manipulate it. Biomechanical cues at the skin’s surface such as contact area and force rate have been thought to help encode compliance. However, no one has directly measured contact area with compliant materials, and few studies have considered compliances softer than the fingerpad. Herein, we developed a novel method to precisely measure the area in contact between compliant stimuli and the fingerpad, at given levels of force and displacement. To determine the method’s robustness, we conducted psychophysical and biomechanical experiments with human subjects. The results indicate that cues including contact area at stimulus peak force of 3 Newtons, force rate over stimulus movement and at peak force, displacement and/or time to reach peak force may help in discriminating compliances while the directional spread of contact area is less important. Between softer and harder compliances, some cues were slightly more evident, though not yet definitively. Based upon the method’s utility, the next step is to conduct broader experiments to distill the mixture of cues that encode compliance. The importance of such work lies in building haptic displays, for example, to render virtual tissues.

I. Introduction

Distinguishing an object’s compliance into percepts of “softness” and “hardness” is crucial to our ability to grasp and manipulate it. Our ability to differentiate compliant materials, with the bare finger, has been investigated in prior psychophysical studies. Such work has been done to inform the design of haptic displays, for example, to render virtual tissues within the human body. However, a prerequisite step is to understand which cues encode compliant stimuli.

One early study found that objects’ softness could be differentiated if their compliances differed by ~13% [2]. Further studies of softness discrimination – under varied conditions including active touch, passive touch, and proprioceptive feedback without tactile sensation – showed that people can differentiate surface compliance with high reliability using only passive tactile information, independent of proprioception [1]. A follow-up study found that subjects could estimate softness independent of kinesthetic/proprioception cues, ramp rate and peak compressional force [3]. Bergmann and Kappers (2009) found that subjects could differentiate compliances with Weber fractions of ~15% and that softness cues might relate to deformation of the skin surface [4]. This cue is in line with Friedman and LaMotte’s proposal that we evaluate softness by the spatial distribution of pressure on the skin, likely through the dynamic change in pressure distribution over time, which is a function of the object [3].

Srinivasan and LaMotte (1995) sought to connect such softness cues in skin deformation with potential peripheral neural codes [1]. Potential softness cues evaluated included net contact force, contact area, pressure distribution within the contact region, and displacement distributions across the skin. During indentation, spatio-temporal changes in force propagation in the skin might be signaled in the recruitment of afferents at the boundary of force propagation. The authors postulated that SAI and RA afferents are particularly suited to encode spatio-temporal softness cues. Relatedly, studies involving haptic displays focus upon both the rate of change of force and contact area as the cues fundamental to displaying different compliances [5–7].

At present, it remains unknown which of these tactile cues are most important in discriminating compliance, or whether the relative weights of these cues are consistent through a range of compliances. Deciphering the compliance of objects either softer or harder than the fingertip may require that we rely upon different cues and employ different strategies. While subjects have been able to describe objects less stiff than the finger as “soft” and those stiffer than the finger as “hard,” [3] objects less compliant than the finger have rarely been studied. One study noted that subjects applied force with the finger more perpendicular to the surface when exploring harder stimuli, possibly to maximize contact area changes [8]. Despite the lack of clarity at present on the cues underlying the discrimination of compliance, it is clear that contact area is widely believed to be a key cue. To our knowledge, however, no one has directly measured finger-stimulus contact area for compliant stimuli, though done in the context of rigid bodies [5, 7, 9].

The objective of the work herein is to introduce and evaluate a novel method to precisely measure the area in contact between compliant stimuli and the fingerpad, at given forces and displacements. We sought to conduct experiments with human subjects to tie observed biomechanical cues with psychophysical discrimination of compliant objects, considering differences both between individuals and for compliance ranges both softer and harder than fingertip skin.

II. Methods

As we could not measure all of biomechanical cues of interest during the psychophysics experiment, e.g., the contact areas at each time step leading up to peak force, we used data taken during the biomechanics experiments to reconstruct the cues subjects might have used in discriminating the compliances. Therefore, our overall experimental procedures followed the four steps outlined in Fig. 1. First, we selected two sets of stimuli, one less and one more compliant than fingertip skin. Second, with these stimuli, psychophysical tests (N=3) ensured that the two stimuli in the less compliant set were discriminable from each other, and likewise for more compliant set of stimuli. These were passive indentation experiments. Third, we developed a novel method to precisely measure the area in contact between the stimulus and fingerpad at given levels of force and displacement. Using this method the four stimuli were indented into the fingertips of the human-subjects (N=5), to different levels of displacement. These were the biomechanical experiments and yielded a series of discrete force to contact area and displacement to contact area relationships. The individuals differed in fingerpad size and finger stiffness. Fourth, we created a procedure for reconstructing the time-course of cutaneous cues. Basically, the contact areas were associated with displacement and force levels at points in time for a continuous indent to the peak force used in the psychophysical tests. Given this data we sought to determine the method’s robustness. In particular, we sought to determine which cues at the fingertip surface (contact area at stimulus peak force of 3 Newtons, force rate over stimulus movement and at peak force, displacement and/or time to reach peak force, directional spread of contact area, etc.) might be different enough such that we could discriminate the two less compliant stimuli, or the two more compliant stimuli.

Fig 1. Outline of experimental procedures.

A) Graphic representation of stimuli comparisons-two “soft” stimuli more compliant than the fingerpad were compared to one another, as well as two “hard” stimuli less compliant than the fingerpad. B) A brief outline of the experimental procedure.

A. Experimental Apparatus: Stimuli and Indenter

We constructed two sets of stimuli, one less compliant or the “hard” set (150 and 120 kPa) than the other more compliant or the “soft” set (22 and 18 kPa). We refer later to the 120 and 150 kPa stimuli as “Hard 1” and “Hard 2,” respectively, and 18 and 22 kPa as “Soft 1” and “Soft 2.” Stimulus kPa was estimated by the mixing ratio of silicone elastomer determined from previous experiments. The sets of compliance values were chosen to be greater or lesser than that of the fingertip; this property was confirmed by determining the stiffness of each stimulus and a human finger using a 6 mm flat-plate indenter indenting at 0.5 mm/s to a force of 1 N. Each stimulus was cylindrical with a diameter of 3.8 cm and height of 1.0 cm, so that its diameter was larger than the fingertip contact. Stimuli were constructed with a silicone elastomer (BJB Enterprises, Tustin, CA; TC-5005) with control of its modulus [10–12]. A small indentation at the center of the surface of each stimulus, approximately 1.0 mm in diameter with a depth of 0.3 mm and imperceptible to the participant, was introduced in the casting process so as to be usable later as a consistent point of comparison between stimuli and across stimulus replications and indentation levels. As well, it enabled a means to decipher the directional spread in contact area (proximal-distal versus lateral-medial).

To indent stimuli into the finger (Fig. 3), we controlled a Newport ILS-100 MVTP Linear Motion Stage with a Newport XPS Motion Controller. A Windows 7 PC running software written in Python 2.7 commanded indentations through an ethernet connection with the motion controller, which directly interfaced with the motion stage. Force was measured by a load cell (0 – 22.4 N range; Omegadyne LCFD-5, Stamford, CO, USA) mounted to a cantilever attached to the motion stage. Force was sampled through an analog-to-digital converter on the motion controller. Each stimulus was screwed directly onto the load cell during experiments. A padded armrest was bolted onto the base of the motion stage to secure the forearm and index finger.

Fig 3.

Indenter setup. One participant’s finger is shown strapped down with Velcro to a solid armrest support platform overlaid with a soft foam. Although not entirely visible, the finger is angled upward (~30 degrees). The cantilever moves the compliant stimulus downward into the finger. The load cell is positioned between the cantilever and the stimulus.

B. Participants

Three participants were included in the psychophysical experiments (mean age = 21.3, SD = 0.6, 2 male, 1 female). Five participants were included in the biomechanical experiments (mean age = 21.8, SD = 0.8, 3 male, 2 female). The three participants in the psychophysical experiments were included in the biomechanical experiments. All enrollees granted their consent to participate. All participants continued to completion and no data were disregarded.

C. Measurement of Contact Area

We developed a novel method to measure the area of fingerpad-stimulus contact upon indentation by compliant stimuli. This method complements one used previously for rigid body stimuli [9]. We directly measured the fingerpad-stimulus contact areas at different indentation levels. An example overview of the process is given in Fig. 4 and detailed below.

Fig 4. Fingerprints and shape/area analysis.

A) Fingerprints stamped onto a sheet of paper after successive indents with the Soft 2 stimulus, scanned into .jpg format. The uncolored area in the middle of the print represents the marker created the small indentation in the stimulus which allowed the experimenter to identify a consistent point between indents. B) A portion of the software tool where the fingerprint is identified, color thresholded, and an outline is determined around the fingerprint. C) The output of the tool: a set of vertices representing the shape of the fingerprint and an area determined by Gauss’s area formula. D) The contact areas for a series of 20 displacements overlaid in sequence.

In specific, washable ink (Studio G Red) was applied to the stimulus before each indent with a stamp pad. After each indent, a sheet of plain white paper was carefully rolled onto the fingerpad in order to transfer the ink to the paper. Between indents the finger was gently wiped with a moist paper towel to remove ink. This process was repeated several times with one sheet of paper at various indentation levels. Afterwards, the sheets of paper were marked with a 5.0 cm line to scale the data, scanned into .jpg format, and processed by custom software (written in Python 2.7). Within the software were identified the points in each fingerprint created by the small indentation on the bottom of each stimulus. The color threshold was adjusted to distinguish the area of the red ink from the background.

We tested two different software methods to analyze the contact areas. The first method was almost entirely automated, and required the use of sheets of paper with a series of printed 5.0 cm by 5.0 cm black boxes to record fingerprints. Each fingerprint recorded during an experiment was placed into a separate box. After scanning an image into .jpg format, a blob-finding algorithm identified the printed black squares on the page. Next, the fingerprint color was thresholded, and the number of pixels above threshold within the bounds of each box was counted. This number was then scaled by the area of the printed box (in pixels) in order to determine the area of the fingerprint. There were a few problems with this method. The number of boxes printed on the page seemed like an unnecessary hindrance on gathering the data. The grooves on the fingerpad created many lightened lines within the fingerprint, which were often missed during thresholding and subsequently not counted in the area calculation. Lastly, there was no way to extract the shape of the contact areas or overlay them as only an area value was calculated. We did not end up using this method.

The second method, which was ultimately used, required user input during the calculation but generated more reliable results and included shape and position information along with area. We used blank sheets of paper with a drawn-on 5.0 cm reference line to record fingerprints with this method. Additionally, we began constructing stimuli with small indentations in their surface, so that a marker would be left on each fingerprint at a consistent reference point. After a series of indentations the sheet of paper was scanned into .jpg format and loaded into software. The software displayed the image and requested that the user identify the marker within each fingerprint. After clicking on each marker, the analyst designated a radius from the marker in which to search for each fingerprint. Fingerprints were stored as a marker location, radius pair.

After selecting each fingerprint on the page, the analyst used a slider to select a threshold value for the fingerprint ink color. This updated the on-screen image with bright red indicating thresholded pixels. The threshold value was modified until the edges of all fingerprints were thresholded. Next the experimenter used another tool in the software to identify the 5.0 cm line in the image. Each end of the line was selected and the software calculated the length of each pixel in centimeters, which was later used to calculate the area of each fingerprint.

For each marker location and radius identified per fingerprint, from earlier, a serial search was conducted to determine the bounds of the fingerprint. Edges of the fingerprint were determined by searching from the top-to-bottom of each circle for transition to thresholded color, then bottom-to-top. This resulted in a series of points, which outlined the fingerprint. This set of points was subtracted from the marker point such that coordinates were consistent between fingerprints from the same stimuli. The final set of points was used to determine an area in pixels using Gauss’s area formula, which was scaled to a physical area in squared centimeters through calculations from the reference line. The final output consisted of a set of coordinates and an area per fingerprint.

D. Experimental Procedures

Two sets of human subjects’ experiments were run. The first, a psychophysical experiment, utilized forced-choice discrimination to evaluate the pairs of less and more compliant stimuli. Using a servomotor affixed to the load cell, we were able to quickly alternate a set of two stimuli during the experiment (8 sec between successive stimulus presentations). A total of 80 trials were run per subject: 40 trials between the two hard stimuli and 40 trials between the two softer stimuli. The 40 trials consisted of a randomized set of 20 trials where the same stimulus was presented twice and 20 trials where different stimuli were presented. Within the hard or soft set, each stimulus was presented first and second in the trial the same number of times. In each trial, the first stimulus was indented into the fingerpad at 2 mm/s and force was measured on the load cell. The indentation speed of 2 mm/s was similar to the range of velocities used in [1], which ranged from 2.4 mm/s to 3.6 mm/s. A limitation of a constant indentation velocity was that the subjects could possibly distinguish the objects based on the total time of indentation; however, a constant velocity allowed us to make comparisons of other biomechanical cues, and informally subjects made no mention of indentation time as a factor in their estimates. The indenter stopped moving when the force reached 3 N and remained still for 1 second. Then the indenter retracted and the next stimulus was presented in the same manner approximately 8 seconds later. After each trial subjects were asked to choose which of the two stimuli was harder.

The second, a biomechanical experiment, utilized our method of measuring the contact area between the stimulus and fingerpad. This was done to generate a series of discrete force to contact area and displacement to contact area relationships. Using this method we indented the four stimuli into the fingertips of the human-subjects, to different levels of displacement. Velocity was controlled at 2 mm/s. A point of contact for each stimulus was determined where the stimulus first made visible contact with the index finger and the subject detected contact. Then, a subsequent set of 15 indentations (5 sets of 3 replications) up to 5 mm was made for each stimulus. We also measured the fingerpad dimensions and finger stiffness of each of the individuals.

E. Procedure for Reconstructing the time-course of cutaneous cues

We created a procedure for reconstructing the time-course of cutaneous cues, for each of the stimuli across each of the individuals. Basically, the contact areas were associated with displacement and force levels at points in time for a continuous indent to the peak force used in the psychophysical tests. We used data from our biomechanical experiment as a “lookup table.” From these reconstructed series of contact areas over a range of displacements and forces, we sought to determine which cues at the fingertip surface (contact area at stimulus peak force of 3 Newtons, force rate over stimulus movement and at peak force, displacement and/or time to reach peak force, directional spread of contact area, etc.) might be different enough such that we could discriminate the two less compliant stimuli, or the two more compliant stimuli.

It was difficult in the biomechanical experiment to determine an exact point of contact, as is nearly always the case when two soft objects contact, so the first step in processing the data was to shift the displacement ranges of each stimulus slightly such that all of the contact areas were equal at time = 0 in the simulated experiment. As velocity was constant at 2 mm/s in the experiment, this meant that displacement ranges of the data would be as close to equal as possible. Next, mean displacement-contact area curves were interpolated linearly between the first and second data points and this interpolation was used to determine the point at which contact area was equal and shift. After all displacement-contact area curves were lined up at t = 0, each displacement range was right-shifted such that the smallest contact area of any stimulus was at a displacement of 0 mm and there were no negative displacements.

After shifting the displacement ranges for each subject, we fit curves to the displacement-contact area and displacement-force relationships for each subject-stimulus pair. A fit of the form a–b*exp(-c*x) was determined for the displacement-contact area curves which seemed to fit the data well. Force-contact area curves were fit with a cubic spline interpolant function if their measured values from the lookup table surpassed 3 N, otherwise with a single exponential function. R² values for the fits ranged from 0.86 to 1. The force-contact area curves were used to approximate the displacement for each subject-stimulus pair at which the force would reach 3 N.

III. Results

A. Psychophysical experiments

We tested the discriminability of each pair of hard and soft stimuli. As Fig. 5 demonstrates, a threshold detection rate of approximately 80% was found for each stimulus pair.

Figure 5. Psychophysical experiments results.

Subjects were able to discriminate correctly ~80% of the time between the two stimuli of the “hard” and “soft” sets at a peak force of 3 N.

B. Biomechanical experiments

Next we examined the biomechanical cues available to the subjects during the psychophysics experiments. First, we collected data as a “lookup table” to construct relationships between force, contact area, and displacement during indentation into the compliant objects. Using the method of measuring contact area with human subjects, we generated a series of discrete force to contact area and displacement to contact area relationships for the four stimuli, a look-up table. Figure 6 outlines the characteristics of each stimulus with displacement-controlled indents over each of the 5 subjects. The top half of the figure plots the data for a single subject, while the bottom half plots data from all subjects.

Figure 6. Raw data from the look-up table: Force, displacement, and contact area relationships for all four stimuli.

In panels A) – C) are shown the data for one of the five subjects, while panels D) – F) plot the data for the aggregate set of five subjects. Soft 1 and Soft 2 refer to stimuli of 18 and 22 kPa, respectively. Hard 1 and Hard 2 refer to stimuli of 120 and 150 kPa, respectively.

In touching upon part of this “lookup table” dataset, force in Fig. 6 was measured at the peak displacement of each indent. Per Fig. 6E, all 5 individuals generated very similar force-contact area relationships for all stimuli, such that the between stimulus differences were for the most part greater than the between individuals differences. In particular, the two “soft” stimuli generated very distinct force-contact area relationships among individuals, despite varying by only 4 kPa. The two hard stimuli were less distinct in this regard. Per Figure 6D, this separation was not observed in the displacement to contact area relationship.

Additionally, measurements were taken of each subject’s distal phalange, displayed in Table 1.

TABLE I.

Summary of the dimensions of each participant’s distal phalange. All units in millimeters.

	Lateral-medial	Thickness	Distal-Proximal
Subject 1	15.3	9.7	23.9
Subject 2	16.3	12.3	25.2
Subject 3	14.4	9.1	24.6
Subject 4	16.0	10.6	28.8
Subject 5	14.3	10.6	24.0
Mean	15.3 ± 0.9	10.5 ± 1.2	25.3 ± 2.0

C. Reconstructing the time-course of cutaneous cues

We then used the “lookup” table constructed from the biomechanical experiments to simulate cues available to subjects in the psychophysics experiment. We used the displacement and force data from the psychophysical experiment, where we indented continuously to 3 N, in an effort to reconstruct the time-course of the change in contact area.

Using the data from Fig. 6 as a lookup table, we simulated continuous indentations to 3 N to recreate what contact areas would have been observed in the psychophysical experiments. This process re-constructed the time-course of cutaneous cues that would have been available to the subjects during the passive psychophysics experiments (Fig. 7).

Figure 7. Reconstructed cutaneous cues based upon the look-up table applied to the psychophysics experiments.

A) Time vs. contact area for all stimuli. Solid dots represent the contact area and time for each individual in the simulated psychophysics experiment at peak force of 3 N. B) Time vs. force rate for all subjects. C) Time vs. displacement for each subjects, assuming constant velocity. D) Time vs. directionality of contact area spread on the fingerpad. Here Soft 1 and Soft 2 refer to stimuli of 18 and 22 kPa, respectively. Hard 1 and Hard 2 refer to stimuli of 120 and 150 kPa.

Differences in the groupings of lines in the plots of Fig. 7 suggest cues that may have been used by the subjects in the discrimination experiments. Fig. 7A plots the contact area curves over time. While contact area over the time course of the movement of the stimulus alone does not appear to clearly separate the stimuli, the contact areas at peak force do appear to be differentiated into discrete clusters. Fig. 7B plots force rate over time and seems to clearly distinguish the two harder stimuli, which were not as differentiated in other metrics, in accordance with prior work [6]. Fig. 7C plots the displacement over time, which has equal slope but different peak displacements for each stimulus. Displacement as a separate cue from contact area may be a useful discrimination cue in active touch experiments. Fig. 7D plots the directionality of contact area spread on the fingerpad. The data suggests that shape of the contact area may not be relevant to this particular discrimination task.

IV. Discussion

The objective of the work herein is to introduce and evaluate a novel method to precisely measure the area in contact between compliant stimuli and the fingerpad, at given levels of force and displacement. While contact area is widely believed to be a key cue, to our knowledge no one has heretofore directly measured finger-stimulus contact area for compliant stimuli, though this has been done in the context of rigid bodies [5, 9]. This method, and the subsequent reconstruction of the time course of the contact areas and cutaneous cues appears to yield between stimulus differences that exceed between individual differences.

In particular, regarding which cues might be used to discriminate these compliant objects, the results indicate that cues including contact area at stimulus peak force of 3 Newtons, force rate over stimulus movement and at peak force, displacement and/or time to reach peak force may help in discriminating compliances while the directional spread of contact area is less important. Such cues align with prior works [1, 5–7].

Based upon the method’s utility, the next step is to conduct a broader set of experiments to distill the mixture of cues that encode compliance. For example, one factor that was not considered was the duration of indentation, which could possibly be a factor between compliances. Here we kept velocity constant between compliances. The duration of indentation could possibly be accounted for in a separate experiment where instead of velocity being held constant, the time to reach 3 N is held constant and velocity is varied. In a prior study by LaMotte, a finding was that velocity did not impact discriminability, so such an experiment at changing velocity might be appropriate [1].

Finally, while not a factor here in this preliminary analysis, the directional spread of contact area may be important for other softness experiments (e.g., lump detection). As well, the force-contact area profiles may be related to the increased force and finger angle used to discriminate hard objects noted by Kaim and Drewing [8].

Fig 2. Stiffness of the stimuli and comparison to data of Srinivasan and Lamotte 1995 [1].

Stimulus stiffness was measured using a 6 mm cylindrical flat-plate indenter and confirmed that the soft set was more compliant than the fingerpad and the hard set less compliant.