Synesthesia in Science and Technology: More than Making the Unseen Visible

Abstract

Much of our science and technology relies on the visualization of complex data, and chemical biology, more than most fields, often deals with complex datasets. There are, however, other ways of making information available to our senses beyond the visual. Rare individuals naturally have sensory crossover, whose synesthesia permits them, for example, to see colors or shapes when hearing sounds or to sense a specific taste with a specific word. Many scientists, technologists and inventors, however, make a conscious attempt to convert one type of sensory-like input to a different sensory output. A laser light show, for example, converts sound to sight; infrared imaging converts heat to sight. Two recent examples of such intentional synesthesia are discussed in this context: sight-tasting and smell-seeing.

Introduction

Ultimately, science is the art of making the unseen visible, but that phrase forgets that there are many other ways to re-examine data and information other than just “visualization”. Because humans are such visual creatures, we naturally assume that the best way to represent all information is in a visual format [1, 2], and indeed many of our instruments are direct extensions of our visual senses, from telescopes to microscopes, from infrared cameras to x-ray imaging. There are, however, many ways of transducing seen or unseen information from one type of sensory-like input to a different sensory output: an intentional synesthesia.

If one thinks of aesthetics as the impact of our senses on our mind (or brain or consciousness) [3], then a fundamental question arises, what happens when we cross the usual sensory inputs? There are, of course, rare individuals whose brains are cross-wired and whose synesthesia permits them to see colors or shapes when hearing sounds or to sense a specific taste with a specific word [4–6]. Synesthesia is a neurological condition in which stimulation of one sensory or cognitive pathway generates an automatic experience in a different sensory or cognitive pathway. There are dozens of variants of synesthesia, although some are more common than others. The most common type of synesthesia are day-coloring (in which specific days of the week are associated with specific colors), color-graphemic (in which letters or numbers or shapes produce colors and simple patterns), and color-auditory (in which some specific heard sound, e.g., voices, music, etc., produces specific colors or textures) [6]. Almost all pairing of senses are possible, however, including sound-touching (feeling an object produces a sound) and even taste-hearing (hearing a sound produces a taste). While full synesthesia is not common, objective measurement (rather than self-reporting) estimates that some synesthetic experiences may occur in even a few percent of the population. To be sure, we all make common use of cross-sensory metaphors (e.g., sweet music, brilliant talk, loud colors, bitter sight, hot jazz, stinky debate, sour face, white noise, etc.), and the importance of similar metaphors in scientific thinking is overwhelming [7].

Science and technology make use of synesthetic concepts to a remarkable, but largely unrecognized, extent. Converting any sort of information to a visual representation, whether a simple graph or a complex flow chart is, of course, the mainstay of science. But such visualization is only one example of intentional synesthesia.

Intentional Synesthesia

Given seven senses (hearing, kinesthesia, sight, smell, taste, thermal, and touch), there are 42 different synesthetic conversions of one input into another output, as shown in Table 1. Some 20 of these possible combinations are easily recognized in our modern technology, and the reader may well think of others. Not so surprisingly, conversion of other senses into sight is universal (i.e., ‘visualization’), but interestingly, so too is our use of conversion from touch to other senses. That is our species’ physiological bias, of course: sight and touch, eye and hand.

Table 1.

Synesthetic technologies convert one input sensation into a different kind of output sensation.

Output:→ Input: ↓	Hearing	Kinesthesia	Sight	Smell	Taste	Thermal	Touch/ Pressure
Hearing	▬▬▬▬		Laser light shows; Voice prints; Sign language				Braille transcription
Kinesthesia	Musical instruments; Laser harp; Motion alarms	▬▬▬▬	Handwriting; Art; Dance; Touchscreen; Motion controllers			Friction match	Tele-manipulator (“waldo”); Virtual reality glove
Sight	Recitation; Sheet music; Soundscapes		▬▬▬▬		BrainPort vision sensor; Color coding of flavors
Smell			Smell-seeing chemical sensors	▬▬▬▬
Taste			Litmus paper & pH dyes; Refractometer	Taste is mostly smell!	▬▬▬▬
Thermal			IR imaging; Liquid crystal thermometers	Cooking & burning; Household deodorants		▬▬▬▬	Memory foam
Touch/Pressure	Speech; Braille reading; Percussion	Handrail; Cane	Force Microscopy	Scratch-n-sniff	Flavor microencapsulation	Faucets; Mood rings; Liquid crystals	▬▬▬▬

A laser light show, for example, converts sound to sight; infrared imaging converts heat to sight; atomic force microscopy transforms touch on the atomic level into visual display. The scratching of microencapsulated fragrances converts touch into smell (i.e., “scratch and sniff”), and the microencapsulation of flavors converts a touch of the tongue into taste. Musical instruments convert kinesthesia (the perception of body position and movement) into sound, a friction match converts movement into thermal output, and even simple handwriting is a conversion of kinesthesia to a visual output. Sign language converts hearing by way of kinesthesia into the visual. Reading out loud and performing from a musical score are conversions of sight to sound, and reading Braille is a conversion of touch to sound. Percussion instruments and even speech itself can be considered a conversion of touch or pressure to sound. It is interesting to think of all the technology and design that goes into the objects that make the conversion, and even the most mundane household items can be viewed as intentional synesthesia: a faucet converts touch to heat or cold, a handrail takes touch to a kinesthetic sense of balance.

The analysis and even the creation of technologies and scientific approaches through the use of intentional synesthesia provide a way to avoid our natural inclination to think primarily in terms of sight and touch. It is an amusing thought experiment, for example, to imagine the technologies that we might have developed if we were evolved from canines (who live and die by their sense of smell) rather than apes (who are much more visually and tactilely oriented). Along these lines, let us now examine two recent examples of intentional synesthesia to illustrate the impact such an approach can have on the creation of new technologies.

Sight-Tasting

The most obvious need for a synesthetic technology is clearly when one sense has been severely compromised: conversion of input from that damaged sense to another functional one can provide a novel approach to sensory prosthetics. Given the importance of sight to us as a species and as individuals, it is no surprise that massive efforts have been made to create prosthetics for the vision impaired. Magnification was known to the ancient Egyptians and other early civilizations. Seneca the Younger (ca. 4 BCE - 65 CE), tutor of Roman Emperor Nero, wrote “Letters, however small and indistinct, are seen enlarged and more clearly through a globe or glass filled with water" and Nero is said to have used an emerald as a corrective lens [8]. Eyeglasses in a recognizable form date to the 1200s in Pisa [9] and bifocals were of course invented by Benjamin Franklin in the 1700s [10]. True visual or retinal prosthesis remains a holy grail for the bioengineering community, and substantial progress has been made in recent times, notably with the Argus II Retinal Prosthesis System from Second Sight Medical Products [11] which uses a 60-microelectrode array implanted in the eye. The difficulties of surgical implants, expense, limited resolution, and severe possible consequences over time, however, remain as major barriers to routine implementation of this approach to prosthetics for the blind.

An extremely intriguing alternative is being developed by Wicab, Inc. of Middleton, WI. that utilizes an intentional synesthesia from sight to taste. This “sight-tasting”technology, invented by Paul Bach-y-Rita in 1998, converts an image from a digital camera into a comparable electrode array that sits upon and stimulates the tongue [12]. As illustrated in Figure 1, Wicab’s BrainPort Vision System consists of a 3 cm by 3 cm electrode array (now above 600 microelectrodes) that sits on the top surface of the tongue), a small belt-held computer, a digital video camera (imaging at 30 fps), and a hand-held controller for zoom and contrast inversion. The camera delivers a coarse image of the scene ahead to the electrode array, which the tongue's nerve cells send to the brain. Minimal training is necessary before a blind person can make use of the tongue information, for example to catch a rolling ball [13–14]. Users often report the sensation as pictures that are painted on the tongue with Champagne bubbles. Participants have been able to recognize high-contrast objects, their location, movement, and some aspects of perspective and depth. The advantages of a non-invasive vision prosthetic are numerous: low cost, no significant side effects, no surgery, and trivial adaptation, especially as improvements in the technology develop.

Figure 1.

The BrainPort vision device from Wicab, Inc. provides a visual prosthetic by converting a digital video image (left) into a microelectrode array that stimulates the tongue (right).

Smell-Seeing

The use of intentional synesthesia can also deliver extensions of our senses and produce useful new technologies. Olfaction, for example, is an exceptionally important to most animals [15, 16], but is woefully underappreciated by us humans, whose sense of smell is notoriously poor. Developing a technology that can provide quantitative olfactory-like characterization is an especially worthy goal, simply because we are olfactorily impaired. Over the past few years, we have developed an “optoelectronic nose” that converts olfactory-like information into a visual output: smell-seeing through the use of colorimetric sensor arrays.

Array based vapor sensing has emerged as a potentially powerful approach toward molecular recognition and the detection of chemically diverse analytes. Based on cross-responsive sensor elements, rather than receptors for specific species, these systems produce composite responses unique to an odorant, in a fashion similar to the mammalian olfactory system [15, 16]. The olfactory system in animals begins with hundreds of olfactory receptors (ORs) in the olfactory epithelia at the uppermost region of the nasal cavities. Unlike the usual lock-and-key enzyme-substrate model of biological specificity, the ORs are highly cross-reactive: there is no one receptor that responds specifically to only one odorant. Instead, it is the pattern of all the OR responses that provide molecular recognition and the ability for the brain to recognize one smell from another.

Previous array detectors for electronic noses [17] have employed a variety of strategies that have used weak chemical interactions (e.g., physical adsorption), including the use of conductive polymers and polymer composites, fluorescent dye doped polymer systems, tin oxide sensors, and polymer coated surface acoustic wave devices. As a consequence of this reliance on weak interactions, most prior electronic nose technology suffers from three severe limitations: (1) the detection of compounds at low concentrations (typically < 1 ppm) relative to their vapor pressures is extremely difficult; (2) the discrimination between compounds within a similar chemical class is limited; and importantly, (3) interference from environmental changes in humidity remains problematic.

The development of new sensor technology faces the dilemma of making sensors that are both increasingly sensitive and increasingly robust. Beyond a certain point, the more sensitive a sensor becomes, inherently the less robust it can be, due to poisoning during use in the real-world environment. The path around this dilemma is the development of disposable sensors, thus unlinking the opposing demands. This permits dramatic improvements in vapor phase analysis using sensors based on interactions with the chemical (rather than the physical) properties of molecules [18, 19]. New compact detectors are needed that are inherently capable of molecular recognition and of distinguishing analytes based on their chemical reactivities. We have recently developed a highly successful approach using nanoporous pigment colorimetric sensor arrays, i.e., an optoelectronic nose [18–25]. These sensor arrays are in essence digital, multidimensional extensions of litmus paper, as shown in Figure 2. These are inexpensive, disposable sensor arrays based on cross-reactive interactions of analytes with an array of chemically responsive dyes: an optical analog of mammalian olfaction. As with the mammalian olfactory system [15–16], it is the composite response of the chemical reactivity of such an array that identifies an odorant or mixture of odorants.

Figure 2.

The colorimetric sensor array is the heart of the optoelectronic nose. An array of 36 chemically responsive nanoporous pigments are printed directly inside a disposable cartridge, which is then imaged by an ordinary digital camera or flatbed scanner.

Chemically responsive dyes change color, in either reflected or absorbed light, upon changes in their chemical environment. There are many classes of such dyes and we wish the array to have as chemically diverse a set of dyes as possible. Among these we include Lewis acid/base colorants (i.e., metal ion containing colorants), Brønsted acidic or basic colorants (i.e., traditional pH indicators), colorants with large permanent dipoles (e.g., solvatochromic dyes or vapochromic materials), and redox responsive colorants, including metal nanoparticle precursors (cf. Figure 2). Our 36-sensor array has evolved over the past several years by statistical evaluation of >300 colorants.

By digitally imaging of each dye spot of the array before and during exposure, the changes in color (as revealed in the color difference maps, Figure 3), we have a quantitative measure of a composite response to volatiles. The interactions between analytes and colorants result in well-defined color changes due to stronger chemical interactions and reactions (i.e., not just physical adsorption), which dramatically improves both sensitivity and specificity compared to any prior enose technology. Importantly the sensor array has been specifically engineering to be insensitive to humidity changes. The difference maps are obtained in real time simply by subtracting the before-exposure image from images during exposure using an ordinary flatbed scanner or digital camera (in the lab) or with a small handheld reader (in the field).

Figure 3.

Color difference maps of the colorimetric sensor array are “molecular fingerprints” unique to each odorant VOC, toxic gas, or mixture at its specific concentration. The difference map is made by subtracting the red color value before exposure from the red color value during exposure, green from green, and blue from blue.

Using the color difference maps, odorants can be differentiated even by eye, and simple pattern recognition techniques give essentially error free recognition of a variety of different analytes. Quantitative analysis is also possible using the 108-dimensional vectors (36 changes in red, green and blue values) that come directly from the difference map; these data sets are easily analyzed by standard chemometric techniques: e.g., hierarchical cluster analysis (HCA).

There are a wider range of applications possible for this smell-seeing technology: detection of toxic gases in the chemical workplace [19–21], identification of explosives [22], quality control of foods or beverages [23], rapid identification of bacteria [24], and even diagnosis of disease by breath analysis (e.g., of lung cancer [25]). Figure 4 shows examples of several of these and the ability of the colorimetric sensor array to identify both single component odorants and complex mixtures.

Fig. 4.

The color difference maps are molecular fingerprints for both individual gases and for complex mixtures. Upper: examples of 15 different toxic industrial chemicals are shown at their immediately dangerous to health or life concentrations. Middle: ten different commercial Arabica coffees can be easily discriminated. Lower: human pathogenic bacteria are rapidly identifiable even between different strains of the same bacterium. All of the patterns are clearly distinct to the eye, even without statistical analysis.

Conclusions

Much of our science and technology derives from taking information or data and creating a visualization of it: from simple line graphs to the latest 3D animations. There are, however, many other ways of making information directly available to our senses beyond only the visual. If aesthetics is the impact of our senses on our mind (or brain or consciousness), then new ways of thinking about information come directly from crossing the usual sensory inputs: an intentional synesthesia. There are, of course, rare individuals whose brains are cross-wired and whose synesthesia permits them, for example, to see colors or shapes when hearing sounds or to sense a specific taste with a specific word. Many scientists, technologists and inventors, however, have made a conscious attempt to convert one type of sensory-like input to a different sensory output. A laser light show, for example, converts sound to sight; infrared imaging converts heat to sight; Braille converts a visual input into touch; and the scratching of microencapsulated fragrances converts touch into smell. Given seven senses (hearing, kinesthesia, sight, smell, taste, thermal, and touch), there are 42 different synesthetic conversions of one input into another output; some 20 of these possible combinations are easily recognized. We have discussed two examples of such intentional synesthesia: the conversion of vision to taste by stimulation of the tongue by a microelectrode array and the conversion of smell to a visual color map by the use of a colorimetric sensory array. The conscious use of synesthesia is a provocative way to look for new technologies or to reexamine old ones.

Highlights.

• Many technologies convert information into a visual representation.

• One may imagine the transduction of information from one of our senses to another.

• Intentional synesthesia among 7 senses gives 42 possible cross-conversions.

• Some 20 of these possible combinations are easily recognized.

• Conversion of olfactory-like input into a visual array output is discussed.