Cutaneous sensory nerve as a substitute for auditory nerve in solving deaf-mutes’ hearing problem: an innovation in multi-channel-array skin-hearing technology

Abstract

The current use of hearing aids and artificial cochleas for deaf-mute individuals depends on their auditory nerve. Skin-hearing technology, a patented system developed by our group, uses a cutaneous sensory nerve to substitute for the auditory nerve to help deaf-mutes to hear sound. This paper introduces a new solution, multi-channel-array skin-hearing technology, to solve the problem of speech discrimination. Based on the filtering principle of hair cells, external voice signals at different frequencies are converted to current signals at corresponding frequencies using electronic multi-channel bandpass filtering technology. Different positions on the skin can be stimulated by the electrode array, allowing the perception and discrimination of external speech signals to be determined by the skin response to the current signals. Through voice frequency analysis, the frequency range of the band-pass filter can also be determined. These findings demonstrate that the sensory nerves in the skin can help to transfer the voice signal and to distinguish the speech signal, suggesting that the skin sensory nerves are good candidates for the replacement of the auditory nerve in addressing deaf-mutes’ hearing problems. Scientific hearing experiments can be more safely performed on the skin. Compared with the artificial cochlea, multi-channel-array skin-hearing aids have lower operation risk in use, are cheaper and are more easily popularized.

Introduction

Skin-hearing technology uses the cutaneous sensory nerves to substitute for the auditory nerve to solve deaf-mutes’ hearing problems, and this process seems to lead to auditory nerve regeneration. At present, wearing a hearing aid or using a cochlear implant is the main solution to deaf-mutes’ hearing problems (Zhang et al., 2008). The hearing aid simply involves sound amplification, so it is suited only to patients with mild hearing impairment (Cai et al., 2008; Li et al., 2009; Zhao, 2009; Cui et al., 2010). By contrast, the cochlear implant is suitable for severe hearing impairment, but the surgery to install it is complex and expensive. Additionally, both the hearing aid and the cochlear implant depend on the intactness of the deaf-mute's auditory nerve (Huang et al. 2008; Liang et al. 2008; Liu et al., 2008; Peng et al., 2008; Min et al., 2009). At present, sign language (Du et al., 2010; Jiang et al., 2010; Zheng, 2010) and lip reading (Wang 2006; Calvert et al., 1997) are generally used in deaf and dumb schools.

Based on these problems, we began to study skin-hearing from the perspective of neural plasticity. In 1998, we applied for the invention patent Device of Electronic Hearing by Contactor (Li, 2000). Li (2006) applied for the invention patent Device of Hearing by Skin with Transformer and has been authorized by the State Intellectual Property Office of the People's Republic of China. Through experiments, we found that discrimination of voice mainly depends on voice frequency (Li and Liu, 2005). In November 20, 2009, we applied for the invention patent Multi-Channel-Array Skin-Hearing Aid (Li, 2010) and acquired the patent in 2011. In November 26, 2009, the journal Nature published Aero-Tactile Integration in Speech Perception (Gick and Deeeick, 2009), which produced a certain impact, but it is still very difficult to make a product to discriminate voice by this method. Skin-hearing involves the stimulation of the skin by the electrical signal based on the sound (Li et al., 2006), thus the deaf-mute can receive sound signals through the skin to achieve the purpose of hearing the sound.

The multi-channel-array skin-hearing aid uses multi- channel band-pass filtering to allow the user to discriminate voices as different skin positions can be stimulated by sound current signals at different frequencies. Multi-channel-array skin-hearing technology has the advantages of not requiring an operator, not relying on intact hearing, low cost, wide applicability and easy popularization over the cochlear implant. We began with the study with the skin-hearing aid using multi-channel band-pass filter technology (Li et al., 2006, 2008, 2009), and a new product, the 12-channel skin-hearing aid, called the High Performance Skin Hearing Aid, was created in our laboratory in 2013. The purpose of this study was to investigate whether the cutaneous sensory nerves can be used to transfer the voice signal and to distinguish the speech signal.

Subjects and Methods

Participants

This study was performed from February 2011 to June 2014 at the Skin-Hearing Research Institute, Shaanxi University of Science & Technology, China. Twenty patients with hearing loss (up to 90 dB), consisting of 11 males and nine females, aged 10–58 years, were included in this study. The study protocol was approved by the Ethics Committee of Shaanxi University of Chinese Medicine, China.

Sound variables

Continuously changing sound frequency; Chinese speech phonemes.

The hearing principles of the human ear

The multi-channel-array skin-hearing aid works according to the hearing principles of the human ear. The human ear, a very complex organ (Figure 1A), can be divided into four components: the auricle, the middle ear, the inner ear and the auditory nerve (Xu et al., 2001; Ma et al., 2004; Chen 2006; Li and Liu, 2009; Zhang et al. 2009). The inner ear is made up of the semicircular canals, the vestibule and the cochlea. The function of the cochlea is to transform the sound signal into a voltaic signal via multi-channel band pass filtering and to analyze the frequency components of sound (Chen, 2006). The cochlea can be unfolded (Figure 1B). There are about 30,000 hair cells distributed on the basement membrane, which is connected to the auditory nerve. The hair cells are responsible for converting sound signals at different frequencies into current pulse signals. The auditory nerve is responsible for carrying the designated frequency signals to the corresponding position in the cerebral cortex (Li and Liu, 2009) (Figure 1D). Although the human ear is very complex, with the exception of the neural pathways that lead to the cerebral cortex, other functions can be performed by electronic technology (Qin, 2004; Kang, 2006), such as audio-electric conversion, amplification, and band-pass filtering. In addition to the auditory nerve, there are other neural channels leading to the cerebral cortex (Figure 1C, D). Good candidates to replace the auditory nerve are the sensory nerves that are all over our body's skin. The sensory center is responsible for processing all kinds of signals sent from dermal sensory nerves, and this region is much larger than the auditory nerve (Li and Liu, 2009).

Figure 1.

Mapping relationship of the human auditory nerve and cutaneous sensory nerves in the human cerebral cortex.

(A) Ear structure. The function of the cochlea is to transform the sound signal into a voltaic signal using multi-channel band pass filtering and to analyze the frequency components of sound (Xu et al., 2001). (B) Schematic diagram of the cochlea expanded. There are about 30,000 hair cells distributed on the basement membrane, which is connected to the auditory nerve (Chen, 2006). (C) The nerve distribution in the skin is very rich and each nerve can transmit the stimulus signal to the cerebral cortex, and the cerebral cortex can distinguish between different stimulus locations (Zhang, 2001). (D) Function zone distribution of the human cerebral cortex. The sensory center is responsible for processing all kinds of signals sent from the dermal sensory nerves, and this region is much larger than that serving the auditory nerve (Li and Liu, 2009).

Skin response to sound signals

The human dermis is rich in nerve endings (Zhang, 2001; Zhang et al., 2009) (Figure 1C). These nerves cannot only receive excitant signals, but can also carry the diversified sensory signals to the cerebral cortex in voltaic form (Xu et al., 2001). However, because the resistance of the human epidermis corneum is very strong, a low power signal of less than 5 V is generally imperceptible. The key to skin-hearing technology is to increase audio frequency voltage through the use of an audio step-up transformer (Li, 2004, 2006), to overcome the strong resistance of the skin stratum corneum.

The experimental environment (Figure 2) can be divided into two sections: the production of the planar electrode (Figure 2A) and the construction of the experimental environment (Figure 2B). The planar electrode is a circular metal piece with a diameter of 2–9 mm that is fixed to a 10 mm × 20 mm insulation board and connected to the output of signal generator by two pieces of wire. In the experimental environment, the planar electrode signals are driven by the signal generator to stimulate the Yifeng point (SJ17). We then measured the voltage critical value at an appointed frequency. There was a wide area surrounding the Yifeng point where the human skin had a response to the sound voltaic signal. Experimental results demonstrated that the frequency range of the human skin responding to the pulse signal was 100–30,000 Hz, with the lowest critical voltage value of about 10 Hz. With the increase in frequency, the critical voltage value is increased by 0.1–0.2 every 1 multiple frequency (Figure 2C) (Li et al., 2010).

Figure 2.

Experiment of skin responding to signals of different frequencies.

(A) Planar electrode and two metal pieces; (B) experimental setup and planar electrode driven by a signal generator; (C) the critical voltage lower limit of the skin responding to pulse signals of different frequencies.

In fact, there are many suitable skin sites that respond to sound voltaic signals, including the forearm, upper arm, ear, thigh, and abdomen. The Yifeng point is a site suitable for a small electrode and the forearm is large enough for the multi-channel skin-hearing aid's electrode array. In our multi-channel-array skin-hearing aid, the sensory nerves in the forearm are used as the hair cells in basement membrane through electronic technology. Cutaneous sensory nerves can feel the sound signals, but can they discriminate voice? We can find the answer by analyzing the voice spectrum first.

Voice spectrum analysis

Using a computer microphone, we can record the general voice message and save it to disk. Shown in Figure 3A is an amplitude-time figure, which can be converted into a frequency-amplitude-time figure by a mathematical method called a Fourier transform (Figure 3B–D). In Figure 3A, it is very difficult to discriminate the difference between the voice signals, but it is easy in Figure 3B because of the frequency information. Figure 3C shows the spectrum analysis chart of the Chinese digit voice ‘1 2 3 4 5 6 7 8 9 10’.

Figure 3.

Experiments of voice frequency analysis.

The spectrum analysis was performed using computer software Adobe Audition 3.0. (A) Amplitude-time of Chinese digit voice ‘1 2 3’. Frequency changes cannot be observed here. It is equivalent to the single channel skin hearing aid. (B) Frequency-amplitude-time of Chinese digit voice ‘1 2 3’. The frequency changes can be observed here. It is equivalent to the multi channel skin hearing aid. (C) Frequency-amplitude-time of Chinese digit voice ‘1 2 3 4 5 6 7 8 9 10’. Each tone can be observed to pass through the spectrum. (D) Frequency-amplitude-time of Chinese phonemes ‘a o e ang eng ing ong j q x’. Each tone has its own spectrum.

As shown in Figure 3B, the energy of the syllable ‘yi’ is mainly located in the region under 500 Hz and between 2,000 and 4,000 Hz. Because it has no initial consonant, there are few changes in the pronunciation process. The energy of the syllable ‘er’ is mainly located in the region under 2,000 Hz. Similarly, because it has no initial consonant, there are few changes in the pronunciation process. There is an evident difference when the syllable changes from ‘yi’ to ‘er’. The syllable ‘san’ is divided into two parts, i.e., the initial consonant ‘s’ and the final vowel section ‘an’, and the initial consonant part ‘s’ has noise characteristics and no stripe. There is an energy vacancy near 5,000 Hz in the final vowel section ‘an’, and also near 2,000 Hz. The final vowel energy distribution structure of the syllable ‘san’ is the same as ‘yi’ and ‘er’. In fact, we have many spectral analyses of voiced sounds, such as the syllables ‘ni wo ta’ (you, I, he), ‘ba ba’ (father), ‘ma ma’ (mother), and the phonemes ‘a o e i u ü b p m f g k h j q x z c s zh ch sh’. We can distinguish voices only through observation of the frequency-amplitude-time figure.

Realization of multi-channel-array skin-hearing aid

The basic principle of multi-channel-array skin-hearing technology is to discriminate voices through making use of derma sensory nerves to feel the frequency-amplitude-time figure. The implementation process can be summarized as follows. First, we made the skin feel the sound current signals via an audio step-up transformer; we produced a single-channel skin-hearing aid via this principle (Figure 4A, B). Second, using mathematical methods such as Fourier transforms, we changed the amplitude-time figure into a frequency-amplitude-time figure (Figure 3B–D), and we found that the key frequency range of the voice was between 300 and 7,000 Hz. Third, we constructed the multi-channel band-pass filter program and successively made band-pass filtering circuitry, such as 4-, 12- and 28-channel systems, to decompose the voice signal to the frequency-amplitude-time mode. We can distinguish voices according to different frequencies to stimulate different skin positions. The frequency centers of band-pass filtering are distributed in the mode of a piano keyboard, shown in Table 1. For the 12-channel system, frequency centers are distributed by 1/3 octave, that is . For the 28-channel system, frequency centers are distributed by 1/6 octave, that is .

Figure 4.

Multi-channel skin-hearing aid and the schematic diagram.

(A) The schematic diagram of the single-channel skin-hearing aid without band pass filter. (B) The single-channel skin-hearing aid, which cannot distinguish a change in frequency. (C) Multi-channel-array skin-hearing diagram. Each channel has independent band-pass filtering function. (D) 12-Channel skin-hearing aid. It can be divided into host, microphone and electrode array. The host size is 113 mm × 54 mm × 17 mm.

Table 1.

Distribution of electrode array

Fourth, we found a suitable skin position. The Yifeng point behind the ear is an encouraging position but the available space is small (Figure 5A). A position inside the arm is better for skin-hearing because of the wide available space (Figure 5B).

Figure 5.

Positions for skin-hearing and the multi-channel-array skin-hearing aid.

(A) Yifeng point (SJ17; behind earlobe); the effect is very good, but the space is too small. (B) Electrode array placed on the inside of the arm. The actual effect depends on the orientation of the electrode array. (C) 12-Channel skin-hearing aid. The electronic circuit is very complex. (D) 28-Channel skin-hearing aid. The greater the number of channels is, the more complex the electronic circuit is.

Fifth, the electrode array must be arranged in order of frequency. Distinguishing voice is slightly correlated with the orientation in which the electrode array is arranged (Figure 5B); the OY direction is better than the OX direction.

Sixth, the electronic circuit is very complex in the 12-channel skin-hearing aid, and it is even more complex in the 28-channel skin-hearing aid. The greater the number of channels is, the more complex the electronic circuits are (Figure 5C, D).

In short, the multi-channel-array skin-hearing aid is not only an innovative method of thinking, it is also an implementation of manufacturing craft.

Test of speech discrimination

Twenty patients with hearing impairment (hearing loss up to 90 dB) were randomly divided into group A and group B. Patients in each group received the same speech discrimination test. First, they repeatedly read the speech signal to activate the perception ability of the skin to the current and to strengthen the memory of skin for sound stimulus. Then, they were tested with Chinese pinyin “a, o, q”; Arabic numerals “1, 3, 8” and Chinese “wo (I), ba ba (father), ma ma (mother)”. Each test speech was read aloud 30 times, and then patients were asked to judge the speech of each test and record it. Finally, the test results were statistically analyzed (Table 2).

Table 2.

Voice test result (n = 10)

Results

Single- and multi-channel experimental results

The development process can be summarized as follows: (1) To make the skin feel the sound current signals through the use of an audio step-up transformer, and a single-channel skin-hearing aid was produced according to this principle in our laboratory (Figure 4A, B). (2) To achieve frequency discrimination, which is divided into five steps (Figure 4C, D), that is, to determine the voice key frequency range (Figure 3), to develop a 12-channel band-pass filter circuit, to find a suitable skin position (Figure 5A, B), and to determine the direction of the electrode array. The ability of voice discrimination is the main performance indicator of skin-hearing aid. In the following experiments (Figure 5B–D), 4-, 12-, 24-, and 28-channel-array skin-hearing aids were used. Multi-channel-array skin-hearing aids can discriminate voices, and the ability to discriminate voices was increased along as the number of channels increased. In the use of the skin-hearing aid, if we feel an obvious difference while stimulating the skin, it is considered to be a success.

Channel experiments

We produced a single-channel skin-hearing aid in 2008 and performed a large number of trials. Deaf-mutes and hearing people were included in the trials. The skin-hearing aid was effective for everyone. Precise trial results are as follows: the majority of sound could be felt through the skin, indicating that the skin can be used to hear the sound. After the artificial sound was gradually changed, only the strength change could be felt, indicating that only the information, strong or weak, about the syllable and the number of Chinese pronunciations could be distinguished. This is because one word has one sound in Chinese. However, this technique did not work in distinguishing the Chinese digit syllables “1 (yi), 2 (er), 3 (san)”. Therefore, the discrimination of voice was not very ideal. Through single channel skin-hearing experiments, we found that the human skin could acquire the sound current signal.

As the filter chip has four channels, we first produced the 4-channel skin-hearing aid (Figure 5C). The four central frequencies were set to 400, 800, 1,600, and 3,200 Hz. Compared with the single-channel skin-hearing aid, the 4-channel skin-hearing aid better discriminated frequency. First, it could easily distinguish the Chinese syllables “1 (yi), 2 (er), 3 (san)”. However, the resolution was not high enough, so the Chinese syllables “2 (er)” and “8 (ba)”, “4 (si)” and “10 (shi)”, “6 (liu)” and “9 (jiu)”, “1 (yi)” and “7 (qi)”, could not be effectively distinguished. The 4-channel skin-hearing experiments underlined the importance of multi-channel band-pass filtering, and we found a better skin position: the forearm skin. Table 1 lists the corresponding relationship between channel number and central frequency. Figure 5B shows the position of the electrodes on the forearm.

12-Channel experiments

To improve the ability to distinguish syllables, we produced 12- and 24-channel skin-hearing aids (Figure 5C, D). We made a 12-channel skin-hearing aid from three independent 4-channel skin-hearing aids (Figure 5C). Compared with the 4-channel skin-hearing aid, the 12-channel skin-hearing aid has superior ability to distinguish the frequency change.

Chinese syllables “1 (yi), 2 (er), 3 (san), 4 (si), 5 (wu), 6 (liu), 7 (qi), 8 (ba), 9 (jiu), 10 (shi)” could all be distinguished. Additionally, the Chinese phonemes “a o e i u ü d t n l g k h r y w” could also be distinguished (Figure 3D). The stronger the ability to distinguish the frequency change is, the stronger the distinction of the syllable is. Thus, the 12-channel skin-hearing aid has greater sound resolution ability.

24-Channel experiments

Next, we made a 24-channel skin-hearing aid. Compared with the 12-channel skin-hearing aid, the 24-channel skin-hearing aid had a stronger ability to distinguish the Chinese phonemes “ang eng ing ong an en in un” (Figure 3D). Finally, using four high pitch speech channels, 4,525 Hz, 5,080 Hz, 5,702 Hz, and 6,400 Hz, we made a 28-channel skin-hearing aid (Figure 5D). The 28-channel skin-hearing aid had a stronger ability to distinguish frequency change compared with the 4-, 12-, and 24-channel skin-hearing aids and could distinguish almost all of the Chinese phonemes. The 28-channel skin-hearing aid could also distinguish Chinese aspirants, such as “ji qi xi zhi zi si ci zhi chi shi” (Figure 3D).

Speech discrimination test

The results of the speech discrimination test are shown in Table 2. Test accuracy rate reached 50% in each group. This level is very significant, i.e., from hardly able to distinguish speech signals to the ability to distinguish speech signals.

Band-pass filter channels and frequency range

Through the above tests, the 28-channel band-pass filter had significantly increased ability to distinguish the speech signal, indicating that speech signals can be distinguished via the skin. The ability of the skin to distinguish the speech signal was closely related to the channel number and frequency range of the band-pass filter. Generally speaking, the greater the number of band-pass filter channels, the wider the frequency range of the band-pass filter, which is very important for the ability of the skin to distinguish speech signals.

Discussion

Multi-channel band-pass filtering technology enables anyone to perceive and distinguish speech signals via the skin, which makes the electronic cochlea no longer the only choice of the completely deaf. The human ear and the auditory nerve are not the only channel for obtaining audio signals, and distinguishing sound signal frequency is the key to distinguishing the voice. Many processes, such as acquiring sound signals, converting from voice signals to current signals and producing frequency resolution, may be accomplished via electronic equipment. Transmitting the sound signal into the cerebral cortex can be accomplished via the cutaneous sensory nerves. With advancements in electronic technology, the deaf-mute hearing problem becomes easier.

We showed in the multi-channel-array skin-hearing experiment that the key point of distinguishing voice is to transfer sufficient frequency-amplitude-time voice messages to the cerebral cortex, to establish a corresponding relationship between the sound frequency and the sensory areas of the brain, and to maintain this correspondence for a long time. The brain's auditory center is not required when distinguishing voice, and over long-term training can be replaced by other regions of the brain. For the skin-hearing aid, sufficient channels can ensure frequency resolution capacity. The effective position of the electrode array on the skin should be kept unchanged. For example, according to the frequency-amplitude-time figure (Figure 3B–D), as long as we read a lot of frequency-amplitude-time information from the voice, we can distinguish speech information by visual observation. In fact, we can now distinguish the voice via the ear sensory nerve, cochlea, skin-hearing aid and the voice frequency-amplitude-time figure.

Multi-channel-array skin-hearing technology has been established in theory and it has also been realized in practice. The channel number and frequency of the band pass filtering settings are very important for distinguishing the speech signal. We set 28 channels in the experiment and increased the frequency of the band pass filter according to the distribution pattern of the piano keys. Compared with previous single-channel filtering experiments, the 28-channel band-pass filter has a stronger ability to distinguish the speech signal. In the single-channel filtering experiments, only the presence of the speech signal could be detected, but it could not be distinguished. By contrast, in multi-channel band-pass filtering experiments, testers could not only detect the existence of the speech signal, but could also discriminate between different speech signals.

In future experiments based on the present results, the frequency points could be set according to octave. That is, 48 channels were used to further enhance the ability to distinguish speech signals. Besides the channel number and frequency of band pass filtering, the test time is also an influential factor of the ability to distinguish the speech signals. The band pass filter is a key factor in distinguishing the voice, and subdivides sound signals to stimulate parts of the human skin. After gaining some experience, users will be more sensitive to the signal of each channel. Therefore, active learning and practice is necessary and contributes to better distinguishing the voice via the cutaneous sensory nerve.