Organic Haptics: Intersection of Materials Chemistry and Tactile Perception

Abstract

The goal of the field of haptics is to create technologies that manipulate the sense of touch. In virtual and augmented reality, haptic devices are for touch what loudspeakers and RGB displays are for hearing and vision. Haptic systems that utilize micromotors or other miniaturized mechanical devices (e.g., for vibration and pneumatic actuation) produce interesting effects, but are quite far from reproducing the feeling of real materials. They are especially deficient in recapitulating surface properties: fine texture, friction, viscoelasticity, tack, and softness. The central argument of this Progress Report is that to reproduce the feel of everyday objects requires chemistry: molecular control over the properties of materials and ultimately design of materials which can change these properties in real time. Stimuli-responsive organic materials, such as polymers and composites, are a class of materials which can change their oxidation state, conductivity, shape, and rheological properties, and thus might be useful in future haptic technologies. Moreover, the use of such materials in research on tactile perception could help elucidate the limits of human tactile sensitivity. The work described represents the beginnings of this new area of inquiry, in which the defining approach is the marriage of materials science and psychology.

Graphical Abstract

1. Introduction

Culture is full of material artifacts that trigger thoughtful or emotional responses, e.g., a violin, a stained glass window, and sparkling wine. These objects and materials derive their timbre, vibrancy, and effervescence from dynamic processes occurring on the microscopic scale. When a human subject engages with these objects, these processes generate action potentials in the afferent neurons of the ear, eyes, nose, and tongue, and ultimately appear in consciousness. Researchers have uncovered many of the ways in which music, visual art, and food elicit emotional and cognitive states. This research is facilitated by technologies which are familiar in everyday life. For example, realistic images and sounds can be created and altered in real time using loudspeakers and display screens. Moreover, the senses of hearing, sight, taste, and smell are localized at specific organs, and thus they are relatively simple to address. In contrast, the structures responsible for mechanosensation—including touch, temperature, pain, proprioception, and other internal sensations—are distributed throughout the somatosensory system and involve many forms of stimuli.^[1–3] For these reasons, haptics—technologies designed to interface with and manipulate the sense of touch—are substantially less developed than those designed to create realistic sensations in hearing and vision, whose sensing organs are localized and respond to a narrower range of inputs.^[4] There thus does not yet exist a “loudspeaker” or “RGB display” for touch that can replicate arbitrary sensations on demand.

Despite the challenges of interfacing with the sense of touch, the field of haptics has made great strides in producing devices that generate compelling and useful sensations for a variety of applications.^[5] In particular, the development of haptic systems for virtual and augmented reality (VR and AR) has drawn together researchers from a diverse range of fields, including computer science, mechanical engineering, and psychology.^[6] Most of this research has utilized well established mechanisms of actuation to stimulate the tactile sense. For example, the use of mechanical vibration, ultrasonic transduction,^[7] and pneumatics (e.g., to provide physical resistance that mimics the solidness of virtual objects) are adept at simulating the bulk properties of objects.^[8] These approaches are, however, limited in their ability to affect texture, tackiness, friction, viscoelasticity, softness, and moisture. These near-surface properties are critically important because they are possessed by nearly all objects in natural environments. In particular, these “squishy” sensations are especially characteristic of biological structures. Materials designed to recapitulate the feel of the biological milieu in real time could have a transformative impact on medical haptics. For example, examination by remote palpation in “healthcare deserts,”^[9] introduction of realistic touch in virtual medical training, and accurate tactile feedback in robot-assisted surgery would be welcome advances.^[10]

While haptics already represents an unusually broad convergence of fields including multiple branches of engineering and cognitive science, one discipline is conspicuous for its absence. Namely, materials science, especially the chemistry of materials. Indeed, all tactile sensations arise at an interface between a soft material (i.e., human skin) and the layer of molecules present at the surface of an object. The “feel” of an object is determined not only by its bulk mechanical properties, temperature and thermal conductivity,^[11] but also friction,^[12] adhesion,^[13,14] and rheological behavior of its surface. Such properties are determined by phase behavior, surface energy, chemical structure, and morphology of the solid material or complex fluid. The ability to alter these properties dynamically for the purposes of generating new tactile sensations could establish “organic haptics” as an important new tool in haptics research and technology (Fig. 1).

Fig. 1.

Organic haptics lies at the intersection of materials chemistry, perception research, and device integration. Image credits: Laure Kayser (“materials chemistry”) and Colin Keef (“perception” and “device integration”).

This Progress Report describes several recent developments at the interface of materials chemistry and tactile perception (organic haptics). At its most basic level, organic haptics is the use of the tools of organic materials chemistry—surface monolayers, polymers, microstructured slabs, and other carbon-based and composite materials—for understanding the sense of touch. In this framework, chemical structure and microstructure of solids are used as the independent variables in psychophysical experiments to measure the sensitivities and thresholds of human tactile perception. (The blanket term for this structure, as described by Whitesides, is the physical organic approach.^[15] This term highlights the commonality of experimental approaches in which one makes systematic changes to a chemical structure in order to understand, for example, solubilities, reaction rates, and other quantities. In the case of organic haptics, one makes small chemical or microstructural changes to learn something about human perception.) The ultimate purpose of this work is to inform the design of new materials whose tactile characteristics can be altered in real time. Examples of such materials are liquid crystal elastomers,^[16] electrorheological fluids,^[17] electrets,^[18] and magnetically actuated materials,^[19–21] among other stimuli-responsive polymers and composites whose properties have yet to be exploited for haptics.^[22–24] The culmination of psychophysical work and materials development is the creation of integrated devices, such as gloves, textiles, and prosthetic appliances which could supply sensations with a level of realism unavailable to “off-the-shelf” components. In a complete system, the haptic devices would be used to communicate bidirectionally with different types of “end effectors” such as robotic arms and objects in virtual reality. Such end effectors would be controlled by the wearable haptic device and would also use sensors onboard to communicate tactile stimuli back to the user.

There are myriad applications for such integrated haptic systems: consumer electronics and gaming, education, training, and VR and AR. Some of the most interesting potential applications of haptics in which organic materials could play a central role are in the biomedical sciences. The invention of “haptic biomaterials” and attendant devices that generate new tactile and kinesthetic sensations could contribute significantly to the biomedical sciences in two ways. First, haptic devices which provide fine spatial control and new modalities of actuation could facilitate a deeper understanding of the structures, neurological processes, and cognitive and perceptual aspects of the tactile sense in humans. Second, devices which produce natural, realistic sensations of fine texture—e.g., beyond tingling, force feedback, and vibration—could be useful in surgical training and human-enhanced robotic surgery,^[25] or in physical therapy^[26] for premature infants,^[27] or patients with neurological disorders (e.g., Parkinson’s disease)^[28] or dysfunction in touch.^[29] While the technological readiness and perceived market need for such technologies has not yet been demonstrated, we note that simpler tactile objects used for training—medical and dental practice dummies or dolls—are already in use.^[30] Even though this article represents only the beginnings of the use of organic materials chemistry in haptics research, we hope we are able to capture some of the excitement which surrounds it. At the very least, we hope to introduce some of the mechanisms of tactile perception, along with the methods of psychophysics, which may be unfamiliar to a materials-oriented readership.

2. Background

Mechanosensation refers to the conversion of mechanical forces (originating inside or outside the body) into signals transduced by the nervous system. In animals, it involves a range of physiological structures and is critical to homeostasis.^[31,32] Mechanosensation includes haptic perception (i.e., the ability to recognize objects through touch) and overlaps in some aspects with interoception (i.e., conscious or non-conscious perception of the internal state of the body, including mechanical forces).^[33,34] Haptic perception can be further subdivided into the tactile and kinesthetic senses. The tactile sense is simply the sense of touch, which involves afferent nerve endings located in the skin. The kinesthetic sense refers to the sense of motion and force. Proprioception, a related concept sometimes used interchangeably with kinesthesia, refers to the awareness of the position of the body. The kinesthetic and proprioceptive senses involve many of the same structures as the tactile sense, but they are located not in the skin but in the musculoskeletal system.^[35] Attempts to manipulate the tactile and kinesthetic senses have used many strategies embodied in a variety of haptic technologies. In the following discussion, we focus on the tactile sense.

2.1. Physiological primer

Human skin contains afferent nerve endings responsible for the detection of tactile signals. In brief, mechanical perturbation of a nerve ending generates an action potential arising from the exchange of cations between the cell and the environment.^[36,37] Action potentials are generated as a result of deformation. Signals from these afferent neurons are perceived as mechanical sensations. A subset of these afferent neurons are terminated in end organs or corpuscles: structures that respond to different modes of mechanical deformation (e.g., pressure and shear). There are four types of mechanosensory corpuscles found in the glabrous (non-hairy) skin of the fingertips.^[38] These corpuscles differ by the depth at which they are found and the types of deformations to which they are the most sensitive. Corpuscles found at the surface, more numerous than those in the deep dermis, have a narrow receptive field (area over which they are sensitive), whereas those in the deep dermis respond to more distant cues. A fast-adapting (FA) mechanoreceptor responds at the beginning and end of a touch (i.e., initial engagement and disengagement with an object) and does not engage in between, whereas a slow-adapting (SA) mechanoreceptor responds throughout the engagement with a surface. The FA mechanoreceptors also respond to mechanical vibrations. Starting at the surface of the skin, the FA-I afferents (terminated in the Meissner’s corpuscles) respond to dynamic vibrations (10–300+ Hz) and fine features on surfaces. The SA-I afferents (terminated in Merkel disks) respond to lower-frequency (0–100+ Hz), static, and dynamic deformation. In the fingertips, the densities of the Meissner and Merkel structures have a lateral distribution similar to that of the fingerprints. In the deep dermis are the FA-II afferents (terminated in Pacinian corpuscles), which respond to transient and high-frequency vibrations (~20–800 Hz), and the SA-II afferents (terminated in Ruffini endings), which respond to stretching and similar static and lower-frequency deformations (0–300+ Hz).^[32,36,39] In addition to these structures, the skin is innervated by free nerve endings and nociceptors, which are activated by stimuli that may cause damage to the underlying tissue. Such afferents, along with the nerve endings that innervate follicles in hairy skin, do not have end organs. An action such as lifting an object will trigger sensations in a wide range of mechanosensory neurons that combine—along with the kinesthetic and proprioceptive sensations—to produce a dynamic impression of an object in consciousness.^[1]

2.2. Tools of haptics research

Given the complexity of the tactile sense, attempts to understand and manipulate it have used many physical, biological, and psychological tools.^[32] For example, the perception of fine texture has been investigated using the principles of contact mechanics applied to the sliding of fingertips along surfaces. Vibrations and shear deformations generated at the interface between the skin and the surface generate action potentials in the appropriate afferent mechanosensory neurons, and the perception can be probed using the tools of neurophysiology and psychology.^[40–43] Haptic technologies^[1] have historically been dominated by devices based on mechanical actuation of motors.^[44–46] The most sophisticated haptic devices available use some combination of motors, electrical signals (e.g., the Teslasuit),^[47] hydraulic or pneumatic actuators,^[48] ultrasonics,^[7] or manipulation of temperature.^[49,50] “Surface haptics” is a subfield whose goal is to control the feeling of an object’s surface (e.g., of the screen of a smartphone or tablet, Fig. 2a–d).^[46,51] Techniques used in surface haptics include those based on vibration, along with electrotactile,^[52] ultrasonic,^[53,54] and electroadhesive^[55] phenomena.^[56] In a new area of research, haptic tools are combined with soft robotic end effectors—grippers, prostheses, and other actuators—for remote applications, i.e., telesurgery (Fig. 2e–i). Other ingredients of haptics technologies are devices that manipulate the kinesthetic sense. Kinesthetic sensations—i.e., those associated with relative position of one’s own body parts as well as exertion—can be generated using conventional (usually bulky) mechanical actuators.^[57]

Fig. 2.

Frontiers in haptic interfaces. (a) Schematic diagram depicting the phenomenon of electrovibration. (b-d) Photograph of participant interacting with touch screens equipped with haptic feedback by electrovibration: compression of a spring, collision of objects, texture of crossbar arrays. Reproduced with permission from ref. ^[51]. Copyright 2011, IEEE. (e) Marriage of soft robotic actuators with haptic feedback for robotic assisted surgery. (f) Digital rendering of patient anatomy. (g) Foot pedals used to switch between modes of interaction with the simulation. (h, i) Construction of soft robotic actuator consisting of concentric tubes. Reproduced with permission from ref. ^[60]. Copyright 2018, Springer.

The sensations produced by existing haptic systems do not, we believe, closely reproduce the “feel” of objects in the natural world—i.e., with similar accuracy with which a digital display can reproduce a visual object or a loudspeaker can reproduce the sound of a musical instrument.^[58] Natural objects derive their surface texture and bulk properties from molecular structure, van der Waals interactions, surface texture, and the phase behavior of matter under specified conditions.^[59] In contrast, the state-of-the-art devices typically apply mechanical or electrical stimulation. Moreover, the limited availability of materials which can convincingly replicate the natural world is a bottleneck in understanding the sense of touch. That is, the psychophysical research community does not ordinarily have access to a level of control of microstructure, thermal properties, and surface energy which might permit isolation of perceptual cues. We believe that without new materials, the advances brought by the combination of psychophysics and mechanical testing may plateau due to an overreliance on the methodologies of continuum contact mechanics. New materials with precise surface properties and that can be actuated on-demand are necessary to explore the range of phenomena responsible for tactile perception.

2.3. Scope

This Progress Report focuses on the intersection of materials chemistry with tactile perception. While we do show some examples of integrated devices, these are primarily intended to highlight the properties of materials, as opposed to compete with the state of the art in haptic actuators. Our principal interest is in the ways these devices can be used to learn about human perception, as opposed to the performance metrics of the devices for any particular technological application (i.e., touch screens). Our focus on organic (especially polymeric) materials and their interfacial properties, phase transitions, mechanical properties, and electrical conductivity is partially a consequence of the expertise of the authors, as opposed to because of any “superiority” of organic materials compared to inorganic ones. However, there are at least six characteristics possessed by organic materials which make them particularly amenable to haptics research. (1) Tactile interaction with virtually every object in almost every environment is mediated by organic media. Even nominally “inorganic” surfaces—of the ocean and desert rocks, and of glass, metal, and ceramic objects in an office—support microbiomes, monolayers (and above) of physisorbed organic molecules, or both.^[61] One has to make a special effort to eliminate organics and to keep these surfaces free of adventitiously adsorbed species. (2) In the case of biological structures, it is not just the properties of the surface, but also those of the bulk that are mediated by soft, organic (usually hydrated) matter. (3) Structures based on carbon exhibit much greater chemical diversity than those based on any other element. Molecular species can be engineered to enable wide range of thermal, mechanical, magnetic, electronic, and optical function. (4) In the context of a wearable device, polymeric materials can be engineered to be stretchable and thus conformable to the skin.^[62] (5) Polymers are amenable to fabrication by printing and molding techniques.^[63] (6) Finally, the well-established techniques of organic synthetic chemistry can be used to make fine adjustments to a molecular structure to affect its function in controlled increments (e.g., substitution of atoms down a group in the periodic table, such as from fluorine to chlorine to bromine, etc., and homologation of alkyl groups from methyl to ethyl to propyl, etc.). This level of control permits systematic elucidation of structure-property relationships.^[15] In particular, this approach may be useful in understanding the ways in which human participants evaluate the thermal, mechanical (especially rheological), and electrostatic properties of materials by touch.

Our interest in this field is motivated mostly by scientific curiosity but also by the potential—even the far off potential—of haptics for healthcare applications. Such applications are the topic of a large and expanding literature. In order to maintain the scope of this Progress Report, we will not cover these areas beyond pointing the reader to the reviews of others. For an excellent and comprehensive review of actuators for haptic displays, we direct the reader to the work of Biswas and Visell.^[6] This work focuses on haptic actuators that deliver a sensation based on force generation that operate by well-established physical principles (pneumatic actuation, vibration, and electrostatics). The use of haptic feedback in VR for medical training has been evaluated in several contexts: opthamology,^[64] neurosurgery,^[65] gastrointestinal endoscopy,^[66] numerous applications in dental and craniofacial surgery,^[30,67] and specific classes of procedures which are routine in many areas of internal medicine, such as needle insertion.^[68] The use of haptic feedback in robot-assisted surgery is similarly of interest. For example, in plastic surgery,^[69] the simulation of the elastomeric properties of various tissues,^[70] and the development of specific tools, e.g., forceps.^[71] Another active area of research central to haptics is the mechanics of complex interfaces. In particular, the dynamic forces present at the skin-surface interface during any engagement with an object have been the topic of many studies,^[72–75] many of which provided inspiration for the work described here.

3. Consilience of materials science and psychology

Our work has been informed by recent reports in which the techniques of psychophysics have been used to explore perceived differences between surfaces. For example, in a recent study by Rutland and coworkers, subjects were asked to scan their fingers across surfaces bearing sinusoidal wrinkles (Fig. 3).^[42] Subjects were able to detect wrinkles with heights as small as 13 nm.^[42] A different study asked human subjects to differentiate smooth surfaces of two materials ubiquitous in everyday life: glass and acrylic plastic, and subjects were able to do so with statistical accuracy.^[43] Similar studies have examined a range of commercially available materials.^[76,77] In our own experiments, we were interested in exploring tactile cues that permitted discrimination of fine texture (especially apparent texture arising from friction mediated by surface forces) and softness. Our approach was to use surface science and microfabrication to isolate variables (surface energy, modulus, roughness, and interfacial contact area) which are otherwise difficult to disentangle using off-the-shelf materials. In the first area, we explored the effect friction arising from differences in surface energy on the ability of human subjects to discriminate otherwise identical surfaces using molecular monolayers. In the second area, we attempted to isolate the effects of indentation depth, contact area, and elastic modulus on the perception of softness using silicone slabs with microstructured relief features. Our (IRB-approved) experiments, described below,^[41,78] used either a two- or three-alternative forced-choice test (i.e., “odd-man-out”) to measure the tactile discriminability of surfaces that differed in surface energy (e.g., hydrophobicity vs. hydrophilicity) or softness. This study measured sensitivity: the goal was not to characterize the threshold difference at which surfaces could be distinguished, but rather to determine whether a difference could be detected at all. Eventually, using dynamic materials which can change their properties in real time, it will become possible to probe tactile thresholds in addition to sensitivity.

Fig. 3.

Exploring the limits of tactile perception: detection of surface wrinkles by human subjects (results from Rutland and coworkers). Topographic images of polymeric surfaces having surface wrinkles with wavelengths of 37 μm and 71 μm (b). (c) Color scale corresponding to the wavelength of the wrinkles. (d) Cartesian plot showing perceived similarities between surfaces (as colored by wavelength of surface wrinkles). Axes were determined by a statistical analysis of subject responses which found that the majority of data was explained by using two unknown, but independent, variables (i.e., Dimension 1 and Dimension 2). The closer together the data points, the more similarly the samples were perceived. (e) Finger friction coefficient plotted against wrinkling wavelength. The similarity in shape between (d) and (e) suggests that perceptual cues arise from physical parameters of the slabs. Reproduced with permission from Ref. ^[42]. Copyright 2012, Nature Publishing Group.

3.1. Perception of frictional differences arising from surface monolayers

We began our work by establishing that human subjects can discriminate (by touch) surfaces that differ by surface energy as influenced by the topmost layer of molecules (Fig. 4).^[41] With the use of a custom-designed apparatus to measure sliding force, we also introduced a new method of quantifying the ability to discriminate surfaces as a function of the velocity and applied pressure of the finger (Fig. 3).^[41] Our approach was as follows. Silicon surfaces were either subjected to plasma oxidation (“SiOH”) or treated with a fluorinated alkyl silane (“FOTS”) and assiduously washed to leave a molecular monolayer. These two types of surfaces were visually indistinguishable and had the same roughness (<1 nm, root-mean-square value). In previous studies which used substantially rougher surfaces,^[42,43,79] the source of the friction was collisions between the skin and asperities at the surfaces. In our experiments, differences in friction arose instead as the result of differences in surface energy (Fig. 4a and 4b). Human subject testing proceeded by giving subjects eight sets of three surfaces, and then asking them to determine which surface was different among the three. Subjects were allowed “free exploration” of the surfaces, and most chose to engage with the surfaces by sliding or stroking with the fingertip (Fig. 4c). To minimize the chances that subjects could differentiate the surfaces by the appearance of moisture or condensation, or the deposition of residue (e.g., dead skin or dried sweat), subjects were blindfolded. To reduce the risk that subjects could identify surfaces based on the production of audible sounds, they were also asked to wear noise-cancelling headphones. A cohort of 15 subjects (120 total trials) correctly identified the dissimilar surface (“odd-man out”, Wald Z test, P < 0.0001) 71% of the time (chance would have produced a value of 33%, Fig. 4d). Since our study, Skedung et al. has performed an analysis of the human ability to discriminate surfaces modified by molecular monolayers of a greater variety of chemical compositions.^[80]

Fig. 4.

Previous results: ability of subjects to discriminate surfaces that differ in surface energy. (a) Structures of hydrophilic (SiOH, top) and hydrophobic (FOTS, bottom) silicon surfaces. (b) Contact angle measurements of water droplets on SiOH (top) and FOTS (bottom). (c) Representation of free exploration (top) and tapping only (bottom) in an “odd-man-out” test. (d) Results of discrimination experiments. Data are mean accuracy and 95% confidence interval. **P < 0.01, **** P < 0.0001. (e) Accuracy of subjects vs. moisture level of the finger. Each data point represents the performance of a single individual (f) Representation of “molecular braille” corresponding to rectangular regions of silicon wafers (2 cm × 8 cm) using 1 cm SiOH and FOTS patterned segments to spell the word “Lab” over three separate wafers. (g) Distribution of successfully decoded bits among subjects: 21 successfully decoded bits corresponds to the correct word. Images and description adapted from ref. ^[41] with permission. Copyright 2018, Royal Society of Chemistry.

We surmised that the feel of the surfaces was controlled by friction and adhesion, which are in principle influenced by surface forces arising from two phenomena: electrostatics (i.e., van der Waals forces) and capillarity (i.e., arising from surface tension of a thin meniscus of water). To eliminate the effect of capillarity, we repeated the experiment with the samples submerged in water. The performance of the subjects improved to 84% (Wald Z test, P < 0.0001). We could not, however, discount a possible training effect, since the same cohort had previously performed the task in the “dry” condition. To examine the possibility that adhesion alone (as opposed to friction) permitted discrimination of the surfaces, we performed a “tap test,” in which the subjects were only allowed to tap the surface, as opposed to stroking them. Subjects obtained an average accuracy of 54%, at a rate significantly better than chance, but worse than with free exploration, in which stroking was allowed. The ability to discriminate between surfaces decreased as a function of skin moisture, as measured with a commercial skin hygrometer, with the responses of two subjects falling below chance (Fig. 4e). Subjects were then asked to “read”—by feel—characters written in the ASCII alphabet recorded as 1s (SiOH) and 0s (FOTS). Subjects could accomplish this task with 45.5% of participants being able to decode a complete three-letter word (Fig. 4f,g).

While the properties of the chemical treatments of the surface used in these experiments were static, it is useful to consider whether other types of surface treatments might be engineered to provide reconfigurable tactile stimuli. Ultrathin silane films (often fluoropolymers) are ubiquitous in the manufacture of touch screens (e.g., 3M Novec coatings). They are used for their ability to resist smudging and abrasion, and to make the screens easier to clean. In future work, it might be possible to design a new, dynamic type of monolayer coating which provides tactile stimuli in addition to its role as a protectant. For example, monolayers can be used to transport charge,^[81] undergo photoinduced isomerization,^[82] participate in redox reactions,^[83] and acquire charge by contact electrification.^[84] Given the sensitivity of human participants to minute forces generated at surfaces, it is possible that exploitation of some of these molecular phenomena may lead to haptic effects unavailable to bulk materials.

To quantify the frictional differences between surfaces under simulated sliding, we built a mechanical apparatus comprising a silicone mockup of a finger attached to a force gauge (Fig. 5a). This mockup was slid across surfaces at different combinations of sliding velocity and mass (in addition to the mass of the mock finger). The force traces produced on SiOH vs. FOTS were oscillatory, i.e., characteristic of stick-slip friction. These oscillations were compared using a cross-correlational analysis. We visualized this analysis using a discriminability matrix (Fig. 5b). This representation was useful for determining, at a glance, the combinations of sliding velocity and downward pressure (mass) that, in principle, would allow human subjects to discriminate the two surfaces. Thus, surfaces that produced similar force traces were deemed “similar” and assigned shades of red; those that produced dissimilar force traces were deemed “discriminable” and assigned shades of green (yellow is intermediate). To support these experimental observations, we also generated discriminability matrices computationally using the semi-empirical analytical model of Ruina.^[85,86] We plotted these results using a similar matrix in Fig. 5c. Again, we found that some combinations of velocity and mass produced identical signals, which suggests that under these conditions (locations within the matrices colored red), subjects would perceive these different surfaces as identical. This result highlights the fact that the feel of an object in part relies on how—i.e., with what force and velocity—one engages with the surface. Needless to say, human subjects do not construct such matrices systematically. Rather, they perform the tactile discrimination tasks automatically, presumably by homing in on the correct combinations of loading force and sliding velocity needed to make the discrimination.

Fig. 5.

Results of friction measurements using a mock finger sliding across hydrophobic and hydrophilic surfaces. (a) Schematic diagram of a silicone rubber “finger.” (b) “Discriminability matrix.” Summary of results for combinations of velocity and mass. Green regions produce different tangential force traces and thus permit discriminability between FOTS and SiOH. (c) The same results but derived using an analytical model of stick-slip friction. Adapted with permission from ref. ^[41]. Copyright 2018 Royal Society of Chemistry. (d) Schematic diagram of fingerprint-inspired relief structures patterned into the surfaces of silicone finger. (e) The presence of these relief structures increases the discriminability of hydrophilic and hydrophobic surfaces for several combinations of velocity and mass. Results shown for relief structures perpendicular to the direction of sliding. Adapted with permission from ref. ^[40]. Copyright 2018 Royal Society of Chemistry.

3.2. Role of fingerprint-inspired relief structures in tactile discrimination

Periodic or quasiperiodic relief structures on the surface of the skin are ubiquitous in the animal kingdom.^[87–89] For example, geckos have setae (foot hairs) which enhance adhesion which allow the animals to walk on walls.^[87] Humans and some primates have fingerprints on the tips of their fingers and toes.^[90] Our interest in tactile perception of chemically modified flat surfaces led us to find an interesting effect of fingerprint-inspired relief structures on the surface of silicone mock fingers. Namely, the tangential force traces (i.e., the friction) were substantially different between mock fingers having “fingerprints” and those which were planar. Moreover, the presence of these “fingerprints” had the effect of exaggerating the differences when mock fingers were slid across hydrophilic vs. hydrophobic substrates (Fig. 3d). When considering the modified discriminability matrix shown in Fig 5e, one sees enhanced discriminabililty—i.e., the plot is green or yellow for most combinations of velocity and mass. That is to say, friction forces obtained on surfaces with different surface energies are easier to discriminate for mock fingers bearing “fingerprints.”^[40] While the evolutionary rationale for the function of fingerprints in humans is controversial,^[13,90,91] the results of our model are at least consistent with a hypothesis which points to a role in tactile discrimination. On firmer experimental ground and of greater technological importance however is the way in which relief structures might be used in tactile appliances in either prostheses worn by humans or in soft robotic grippers which might in the future perform tactile discrimination based on friction.

4. Perception of softness

The descriptors “soft” and “hard” when describing the feel of an object are intuitive. The mechanical properties of materials which govern this perception, however, are not well understood. The definition from materials science says that an object is “soft” to the extent that it deforms a certain depth (“indentation depth”) and spread across the indenter by a certain area (“contact area,” Fig. 6a). These two parameters are difficult to separate experimentally, because the indentation depth and contact area change with a constant rate for most off-the-shelf materials with isotropic mechanical properties. Hertzian contact models (consisting of deformable indenters impinging upon deformable slabs) applied to conventional materials has therefore not led to a straightforward description of the ways in which the properties of materials affect the perception of softness.^[92,93] Moreover, psychophysical experiments which strap the hands of participants in restrictive apparatuses represented obtrusive and unrealistic scenarios. Our goal was thus to develop a prescriptive model for the contributions of indentation depth and contact area such that it would be possible to “dial-in” a desired level of softness in engineered objects. Eventually, such a model could inform the design of active materials in which these two properties could be adjusted in real time. We thus engineered an array of microstructured slabs of PDMS to “dial in” the indentation depth, contact area, or both, when pressed into with a fingertip. The indentation depth was varied by controlling the thickness (Fig. 6b). (Because of the confinement of deformation, thin elastomeric slabs have higher effective stiffnesses than thick ones.) The contact area was tuned by fabricating arrays of cylindrical pits into the PDMS (Fig. 6c–e). For these samples, the elastic modulus could not uniquely determine the contact area nor indentation depth at a given applied force, but we were nevertheless able to control it using the ratio of base-to-crosslinker of the prepolymer.

Fig. 6.

Use of silicone slabs to isolate the parameters that affect the perception of softness. (a) When a finger presses into a soft object, it produces an indentation depth (δ) and contact radius (a, contact area = πa²). These parameters can be separated from the intrinsic elastic modulus of the silicone by (b) controlling the thickness of the slab (thinner slabs have higher effective stiffnesses and thus reduced indentation depth) or (c-d) micropatterning pits into the silicone slab (to reduce the contact area). (c) Pits were fabricated by creating pillars on a wafer, pressing the structured side of this wafer into a silicone prepolymer, (d) and curing at 60 °C. Viscoelastic tack of the surface of the cured silicone is reduced by oxidative crosslinking with ultraviolet/ozone. (e) The true contact area excludes the void regions in the locations of the pits. Description and drawings reproduced with permission from ref. ^[78]. Copyright 2019, American Association for the Advancement of Science.

4.1. Human-subject tests: forced-choice and magnitude estimation

We asked participants to perform two psychophysical tasks with the slabs: (1) a two-alternative forced-choice test (Fig. 7a–d) and (2) a magnitude-estimation test (Fig. 7e). As opposed to previous studies in which the hands of participants were immobilized, we allowed a cohort of subjects free exploration of the slabs. We found that reducing the sample thickness or the effective surface area were both effective in reducing the perceived softness of the sample. Moreover, these results demonstrated that consideration of the intensive material properties alone—i.e., the elastic modulus—is insufficient to predict the perceived softness of an object. Forced-choice and magnitude estimation data (fit using a Bradley-Terry model and a linear model, respectively) were compared to the predictions made by several modified Hertzian models of deformation. The theoretical model most predictive of human data (as scored by Akaike information criterion) provided a relationship between the perception of softness and the material properties of the sample, such as sample thickness, elastic modulus, and the effective surface area.

Fig. 7.

Understanding the effect of indentation depth (δ) and contact area (πa²) on the perception of softness. Using the slabs fabricated with the procedure outlined in Fig. 7, δ and πa² could be tuned independently by modulating the thickness (h), modulus (E), or microstructuring of elastomeric slabs. (a) Result of 2-alternative forced-choice comparison of 9 slabs that varied by the parameters shown in Fig. 6. Samples coded by color and shape as described in ref. ^[78]. (b) Comparison of results of two types of psychophysical tests: forced choice and magnitude estimation. Axis labels for “Slab” correspond to those of the forced-choice test. (c) Analysis of results using the Akaike Information Criterion show that a modified Hertzian model with a rigid finger (red dashed line) is the best. Reproduced with permission from ref. ^[78]. Copyright 2019, American Association for the Advancement of Science.

The net result of this analysis was equation 1, below, which prescribes the modulus and deflection when indented, and surface porosity which place all objects used in this study on a 1D scale of softness.

$$\mathit{\text{Softness}}=-8.4+1127\left[{\mathrm{m}}^{-\frac{1}{2}}\right]\mathrm{ }\sqrt{\delta }+1.84\times {10}^6\left[{\mathrm{m}}^{-2}\right]\pi a^2$$ (1)

Where δ is the indentation depth, a is the contact radius, πa² is the contact area, and m is meters.^[78] For example, for a thick sample, the equation predicts that an approximately three-fold reduction in the elastic modulus will result in a sample that is perceived as twice as soft as the original. Among other insights, this study also suggests that “softness” is mathematically transitive (by the lack of blue boxes above, or red boxes below the diagonal line in Fig. 7a) and therefore, the perception of “softness” is a fundamental tactile sensation that is not composed of other sensations. Furthermore, we found subjects were sensitive to relief structures with much smaller spacing than that of the mechanoreceptors in the skin. Even though the relief structures were too small to be felt individually, a collection of relief structures on a surface caused subjects to perceive the surface as harder than a smooth surface (provided that the relief structures deformed minimally relative to the deformation of the bulk). In other words, certain types of relief structures have the effect of reducing the perceived softness (increasing the perceived hardness). This unexpected effect may be due to the production of inhomogeneous strain fields into the depth of the skin due to the relief structure of the surface of the slab. One interesting limitation of this study is that it was conducted in English with participants for whom we expected “softness” was understood as the opposite of “hardness.” In other languages—or for other English speakers—it is possible that the idea of “softness” is more closely associated with “furriness.” However, we expect that the perception captured by the meaning “soft”—as the opposite of “hard”—would remain a fundamental sensation regardless of linguistic limitations. As was the case in our studies involving molecular monolayers, our investigation of “softness” too used static materials. Ultimately, we hope that purpose-synthesized stimuli-responsive elastomers cast into relief patterns^[94] might be used to change the perceived softness of haptic surfaces in real time, e.g., by changing their geometry.

5. Conductive polymers for electrotactile stimulation

Electrotactile stimulation is a type of sensory substitution in which an electrical signal is used to generate an action potential in a mechanosensory neuron.^[95] That is, the electrical signal applied at the surface of the skin (transcutaneous) “bypasses” the mechanical response of the mechanosensory corpuscles. While electrotactile stimulation is not currently a method of generating sensations characterized by a high degree of realism, aspects of their performance can be improved, and they have uses in a number of healthcare applications. For example, they are currently used as transcutaneous electrical nerve stimulation (TENS)^[96] devices available over-the-counter and are primarily used for portable relief of muscle aches.^[97] In an electrotactile device, two electrodes are required to be in contact with the skin where sweat acts as the electrolyte. Typical electrodes used in this type of stimulation are metallic thin films, which suffer from two problems: (1) high electrical impedance and (2) mechanical rigidity. High impedance requires high—i.e., potentially harmful—voltages, while rigidity precludes intimate contact between the electrodes and the microscale asperities of the skin. It also presents a challenge to integrate into skin-conformable, stretchable form factors. Recently it was discovered that coating metallic electrodes with a hydrated conductive polymer, such as the commercially available poly(3,4-ethylenedioxythiophene):poly(styrenesulfonate) (PEDOT:PSS),^[98] has significantly reduced impedance compared to metallic thin films.^[99] Reduced impedance arises from the fact that PEDOT:PSS exhibits both ionic and electronic conductivity, and also because of its three-dimensional structure. A high surface area increases the number of ions that can be exchanged with the electrolyte (e.g., water in the polymer and at the skin surface) and reduces the impedance. However, commercially available PEDOT:PSS is stiff and rigid.^[100] Recently, we have designed a new type of polymer that meets the criteria of (1) low impedance and (2) mechanical softness.^[101]

5.1. Intrinsically stretchable conductive polymer for electrotactile interfaces

The conductive polymer PEDOT:PSS is a polyelectrolyte complex comprising two components: an insulating scaffold of PSS decorated by short segments of conductive PEDOT. It its unmodified state (e.g., that obtained commercially), it is mechanically rigid. Our strategy for rendering this material soft and stretchable involved synthesizing a block copolymer of PSS and an acrylic polymer with poly(ethylene oxide) side chains (Fig. 8). This material has the structure of a bottle brush (reminiscent of the slippery proteoglycans—e.g., aggregan—found in cartilage) and renders the scaffold soft and stretchable. When we oxidatively polymerized the PEDOT around this scaffold (Fig. 8a), we produced a material which was exceptionally tough and stretchable among conductive polymers. The polymer was capable of delivering electrotactile stimuli (Fig. 8b) in a stretchable form factor (Fig. 8c).

Fig. 8.

Synthesis and properties of intrinsically conductive polymer. (a) Details of the synthetic route used to generate the stretchable conductive polymer, PSS-block-PPEGMEA-block-PSS. (b) The material contains no chemical additives and is thus safe to make contact with human skin. (c) Stretchability of the material will permit microscale contact to human skin. The mixed conductivity of the material (ionic + electrical) may permit lower operation voltages in haptic stimulation using electrotactile effects. Adapted with permission from ref. ^[101]. Copyright 2018, American Chemical Society. (d) Exploded view of a device combining electrotactile and pneumotactile functionality. A stretchable conductive polymer blend consisting of PEDOT, tosylate (OTs), and polyurethane (PU) is used to cover a 2 × 2 array of air-filled blisters in silicone elastomers. The diagram to the right is a finite element model showing the peak strains upon inflation of the blisters. (Unpublished result.)

5.2. Combination of electrotactile and pneumatic actuation: “electropneumotactile” arrays

Multimodal haptic actuators—those capable of delivering more than one type of simulation—are crucial to delivering nuanced sensations to the skin in real time. One type of “conventional” actuator (as opposed to one based on stimuli-responsive organic materials) is a pneumatic blister. These devices comprise air pockets in elastomeric slabs which, upon inflation, can produce sensations of pressure or vibration on the skin, depending on the frequency of actuation. The advantage of an intrinsically conductive polymer is that it can coated on the surface of an array of pneumatic blisters to deliver electrotactile signals in concert with pneumotactile ones. We call array of these mixed-mode actuators “electropneumotactile” devices (Fig. 8d). In preliminary experiments, we used a conductive polymer blend consisting of PEDOT and elastomeric polyurethane (PU) with the positive charge of PEDOT balanced by the tosylate (OTs) anion. In these devices, both forms of stimulation were co-localized to four pixels arranged in a square array. In a series of psychophysical discrimination tasks, it was possible for participants to discern the locations of both electrotactile and pneumotactile signals emanating from the array with accuracy substantially higher than chance.

5.2. Ionically conductive organogels for “ionotactile” stimulation

Our group has also examined the use of ionically conductive organogels for low-impedance, soft conductive electrodes for electrotactile (“ionotactile”) stimulation.^[102] The organogel was similar to a conventional polyacrylamide hydrogel but with the water replaced with a saturated solution of sodium chloride in glycerol to prevent evaporation. In this work, we fabricated a thimble-like device containing three ionically conductive “pixels” (Fig. 9a). Fastening this device around the finger (Fig. 9b) permitted acquisition of the sensation curves shown in Fig. 9c. Electrotactile stimulation uses an AC current and is perceived as a mild tingling at low voltages (black curve) and becomes uncomfortable at high ones (red curve). Unfortunately, the organogel was not amenable to microscale patterning (the “pixels” had to be cut from a larger slab with scissors and then manually placed in the silicone thimble). The use of advanced patterning techniques may make it possible to measure the “resolution” of perception (e.g., a two-point discrimination test), and also to generate propagating, 2D sensations.^[101]

Fig. 9.

Electrotactile stimulation using ionically conductive organogels embedded in a thimble-like device made of silicone rubber. (a) Normalized mass v. time for the ionically conductive gel containing either water (where the mass loss is significant) or glycerol (where the mass loss is minor). (b,c) Photographs of the device. (d) Sensation curves showing the source voltage required for stimulation for a range of frequencies. The black curve is the lower limit of sensitivity; the red curve is the limit of comfort. Error bars show the standard deviation between four subjects. Adapted from ref. ^[102]. Copyright 2018, American Chemical Society.

6. Kinesthetic simulation using organic materials

In addition to the tactile sense, haptics also comprises the sense of movement and physical forces (kinesthesia). Such awareness is made possible by afferents found in muscles and connective tissue.^[35] Manipulation of the kinesthetic sense using a wearable device could be useful in a wide variety of settings, such as in VR and AR, training, and therapy.^[26,103] In the area of rehabilitation, patients exhibiting reduction in sensation (e.g., as the result of stroke or injury) may benefit from the use of kinesthetic devices.^[103,104] Most types of kinesthetic appliances are non-portable, which require placing the hands in bulky stirrups or take the form of gloves with large inflatable pneumatic attachments with their attendant pumps.^[105–108] In this work, we developed a wearable, highly portable, composite of a textile and acrylic thermoplastic in a glove in which the stiffness could be changed by heating and cooling with embedded thermoelectric devices.

6.1. Feedback using a thermoplastic-textile composite

In an example from our laboratory, kinesthetic feedback was generated by changing the stiffness of a Lycra fabric modified with a thermoplastic polymer, poly(butylmethacrylate) (PBMA, Fig. 10a).^[109] Changing the temperature of this material in the viscinity of its glass transition temperature (T_g) with thermoelectric (TE) devices produced large changes its mechanical properties (e.g., a two-order-of-magnitude decrease in storage modulus). If used in an article of clothing, the textile composite can be repaired after accidental stretching in its stiffened state by heating above its T_g (Fig. 10b,c). The temperature—and thus the phase of the textile-polymer composite—can be controlled by means of thermoelectric devices attached above and below the knuckles. Psychophysical experiments revealed that the stiffened or flexible state of the textile can be perceived by human participants. As an example of a way in which this type of device can interface with robotic end effectors, the glove was fitted with flex sensors to control a robotic finger, on whose tip there is a pressure sensor (Fig. 10d). When the robotic finger made contact with a rigid obstacle, a signal from a pressure sensor on board the finger was sent to the glove to cool the thermoplastic component of the textile and thus provide the sensation of physical resistance to the user. While the reaction times were high, on the order of 10 s, this work highlights the ability of human participants to sense phase changes in polymeric materials. We are currently investigating strategies to increase the reaction time through the layout of the device or thermal management.

Fig. 10.

Overview of kinesthetic glove and self-healing properties. (a) Schematic drawing illustrating the use of variable stiffness material to produce kinesthetic feedback. Active heating and cooling of spandex/PBMA/DEP above and below the T_g causes a change in stiffness of warp-knit spandex infiltrated with PBMA/DEP, shown in the scanning electron microscope (SEM) image. Concept of tuning the T_g of PBMA to be near the temperature of the ambient environment (~23 °C) or the skin (~32 °C) generates large changes in stiffness with minimal changes in temperature. (b, left) Response times of perceived stiffening as a function of voltage applied to the thermoelectric devices (n = 2 subjects, 30 samples per reported mean and s.d.). (b, right) Comparison of perceived stiffening times of the kinesthetic glove in “pristine” (black), “damaged” (blue), and “healed” (red) conditions (90 °C for 17 h) (1 subject, 10 samples per reported mean and s.d.). (c) SEM images of “pristine,” “damaged,” and “healed” spandex/PBMA/DEP samples. (d) Upper portion, two-way communication of kinesthetic glove with robotic finger. Lower portion, still images of each stage of the demonstration; softening (green), stiffening (purple), and re-softening (pink). Images and description adapted with permission from ref. ^[109]. Copyright 2019, Elsevier.

7. Device integration

The ultimate goal of this work is to combine the knowledge produced by the psychophysical tests and produce integrated devices using organic actuators. As an example of a work-in-progress in our own laboratory, we recently set out to build a wireless glove capable of transmitting tactile sensations. (Images from this preliminary work can be seen in Fig. 1 under “human-subject testing” and “device integration.”) The glove is instrumented with commercial sensors for strain at the knuckles, and for motion of the hand at the wrist. The device was built using a previous design which used sensors based on conductive polymers to convert gestures to text in a rudimentary form of fingerspelling.^[110] The new glove—for the “transmission of touch”—is capable of controlling two different end effectors: a “virtual complement” (a hand in VR) and a “physical complement” (a robotic arm). Signals from the virtual environment, or those transduced by commercial pressure and temperature sensors located on the fingertips of the robotic arm, are transmitted wirelessly to the glove. In a series of preliminary psychophysical experiments, it was possible for untrained participants to distinguish eight different slabs of material in VR that differed by softness, temperature, and texture. In these experiments, tactile sensations were generated using combinations of stimuli-responsive materials developed by us and off-the-shelf actuators. Eventually, all of the signals coming into the glove will be transduced to the user using stimuli-responsive organics—haptic biomaterials—acting in concert. Details of this work are forthcoming.

8. Conclusions

The development of stimuli-responsive organic materials for human-machine interfaces has the potential enable the next generation of haptic devices. The ability of existing haptic elements—e.g., pneumatic actuators, vibrotactile devices, and simple electric heaters—to generate the range of sensations felt in the real world is limited. This article has argued that in order to replicate the range of elastic modulus, softness, roughness, thermal conductivity, coefficient of friction, static charge, and surface energy possessed by natural materials—and to reconfigure such properties in real time—control over molecular structure is required. For example, our investigation of the mechanism by which human participants distinguish hydrophobic and hydrophobic substrates demonstrated that participants are sensitive to differences between materials which arise from a single layer of molecules. While the demonstrations in this work are quite simple, they represent the first step toward a new generation of haptic tools based on organic materials.

While organic and composite materials have many tactile properties that are difficult to replicate in traditional haptic actuators (e.g., those based on displacement), they should be treated as complementary to existing systems. It is likely that in order to realize all tactile sensations of which a human is capable of experiencing in the form of a future haptic display, medical practice dummy, or glove, clothing, or other wearable appliance for VR and AR, many types of haptic actuators will be needed that act in concert. The notion of the “RGB of touch” is useful to frame the goal of this research, but it is highly likely that a haptic system capable of recapitulating the full gamut of touch sensation will require many more than three modalities (i.e., pixel types, to extend the analogy). Indeed, organic, polymeric, and composite materials have significant drawbacks. For example, because of their ionic character, conductive polymers tend to be hygroscopic. Moreover, thermally activated modes of charge transport in these materials produce variations in conductivity with temperature, which could be a challenge to overcome when used in environments with disparate conditions. While it is likely that stretchable forms of encapsulation may be able to solve such problems of stability in some circumstances, other aspects of organic—particularly polymeric—materials are less obviously soluble. That is, polymers and other structured fluids are viscoelastic, and actuators based on them tend to have poor kinetics. Such materials cannot generally respond as quickly (or with as high frequencies) as haptic devices based on pneumatics, motors, and piezoelectric devices. It should be possible to tune the molecular structure using synthetic chemistry, or the device layout using mechanical modeling, to lessen the effects of viscoelasticity. Further innovation will thus require a marriage of molecular and continuum mechanics.

Indeed, one of the more unique aspects of organic haptics is its interdisciplinarity. That is, the toolkits of materials science and psychology are not commonly applied toward a common purpose. While the role of materials science in organic haptics is obvious, the tools of psychology are required for (1) experimental design, (2) workup of data to validate significance, and (3) understanding of how tactile interaction with a physical object becomes an object in consciousness. Physical scientists may feel uncomfortable with the qualitative nature of subjective experience, along with the scatter in data that is often obtained. In such cases, however, the statistical methods developed in psychology become crucial to tease out actual perception from statistical anomalies. Moreover, physical scientists and engineers must work with experts to design careful studies of human participants and receive approval from their Institutional Review Boards (IRBs). We encourage recruiting of participants who have no familiarity with the topic, as is typical in psychological research. That is, the use of coauthors as participants inherently introduces bias. It is also critical to work with materials already used in biomedical devices or which pose no risk of exposure when touched.

An inviting long-term application for organic haptics is as a tool to understand mechanosensation in humans more broadly. That is, touch is unique among the five senses in that it is not localized to a specific organ (e.g., the eye) and involves a broad range of structures (e.g., tissues, corpuscles, and ion channels) distributed throughout the somatosensory system. Because of this complexity, there are many aspects of mechanosensation—including interoception, proprioception, and pain—that remain poorly understood because we currently lack the tools to probe them. Our key supposition is that these tools are best designed using soft, active materials for unobtrusive integration with biological systems—i.e., “haptic biomaterials.” In the context of medical touch (i.e., for robotic surgery, virtual medical training, haptic dummies, and telemedicine), haptic biomaterials are unique among all biomaterials because their interactions with biological systems are dual facing: (1) they must mimic aspects of biological tissue (e.g., of the “patient”) and (2) must do so by interacting with the somatosensory system (e.g., of the “clinician”). These materials would serve the same function for medical touch as the RGB display serves for vision and the loudspeaker serves for hearing. While such a technology is far from being realized and the medical use cases are not yet defined, we believe that the chemistry of stimuli-responsive organic materials could play a central role in its development. This technology could usher in the era of a Star-Trek-inspired “medical holodeck” and transform aspects of medical training, telemedicine, prostheses, and assistive devices.

Corresponding author biography

Darren J. Lipomi is a professor in the Department of NanoEngineering and programs in Chemical Engineering and Materials Science at the University of California, San Diego. He earned his PhD from Harvard University in 2010 in the laboratory of Prof. George M. Whitesides and was a postdoctoral fellow at Stanford University in the laboratory of Prof. Zhenan Bao from 2010–2012. He is the recipient of the Air Force Office of Scientific Research Young Investigator Award, the National Institutes of Health Director’s New Innovator Award, and the Presidential Early Career Award for Scientists and Engineers. His current research focuses on the mechanical properties of electronic polymers for applications in energy and healthcare.