Soft eSkin: distributed touch sensing with harmonized energy and computing

Abstract

Inspired by biology, significant advances have been made in the field of electronic skin (eSkin) or tactile skin. Many of these advances have come through mimicking the morphology of human skin and by distributing few touch sensors in an area. However, the complexity of human skin goes beyond mimicking few morphological features or using few sensors. For example, embedded computing (e.g. processing of tactile data at the point of contact) is centric to the human skin as some neuroscience studies show. Likewise, distributed cell or molecular energy is a key feature of human skin. The eSkin with such features, along with distributed and embedded sensors/electronics on soft substrates, is an interesting topic to explore. These features also make eSkin significantly different from conventional computing. For example, unlike conventional centralized computing enabled by miniaturized chips, the eSkin could be seen as a flexible and wearable large area computer with distributed sensors and harmonized energy. This paper discusses these advanced features in eSkin, particularly the distributed sensing harmoniously integrated with energy harvesters, storage devices and distributed computing to read and locally process the tactile sensory data. Rapid advances in neuromorphic hardware, flexible energy generation, energy-conscious electronics, flexible and printed electronics are also discussed.

This article is part of the theme issue ‘Harmonizing energy-autonomous computing and intelligence’.

1. Introduction

Skin is the largest organ in the human body (overall surface area of 1.5–2.0 m²) [1], which houses a network of distributed and energy-aware receptors at various depths in the soft tissues [2–4]. These receptors (mechanoreceptors (pressure/force), nociceptors (pain) and thermoreceptors (temperature)) enable us to feel and perceive various contact parameters [2–4] and give us the most vital of the five essential senses (sight, hearing, touch, smell, taste). Based on the stimuli the mechanoreceptors respond to, their classification as SAs (slow adapting) and FAs (fast adapting) is well known [2–4]. In daily tasks such as lifting an object (figure 1a), the FAs are active only during the phase transitions (e.g. beginning of contact phase or lift off phase, etc.), whereas the SAs are active throughout the tasks, i.e. even under static or quasi-static conditions (figure 1b) [2–7]. The human skin therefore responds to both dynamic and static or quasi-static stimuli through FAs and SAs [2–7]. Since SAs continue to generate spikes throughout the static or quasi-static events (figure 1b), the neuromorphic approach in eSkin is fundamentally different from the event-driven approaches that are widely explored today in context with vision and other sensory modalities [2–4,8–14]. This neural computing aspect of human skin (hereafter referred to as ‘skin’ only) is discussed later in this section. As per some rough estimates, more than 45 K SAs and FAs are distributed in the skin (figure 2) and the data generated by these large number of receptors during a physical interaction are handled through local computation and packing the information in the form of spikes or action potentials [10,15–20].

Figure 1.

Tactile sensing system in humans. (a) A person performing the simple task of pick and place. It also shows the zoomed in portion of human skin with various mechanoreceptors at different levels [3] along with the transmission of tactile signal from the fingertip to brain in limited bandwidth. Inset in (a) shows the summary of processes involved during tactile signal. Adapted from [3]. Copyright © 2012 Springer. (b) Sequential coordination and discrete control events [5]. Copyright © 2009 Elsevier Ltd. (Online version in colour.)

Figure 2.

Distribution of mechanoreceptors in the human body. Inset shows the glabrous area of the hand (100%), which corresponds to 1.2% of the total skin area [15–19]. Adapted from [17]. Copyright © 2017 Navaraj et al. 2017. (Online version in colour.)

The local computation at the receptor sites in the skin reduces or scales down the amount of tactile data to be transmitted to higher computation/perceptual levels. The lesser data handling also reduces the energy requirements and thus in a sense exemplifies the harmonized energy and computing [10,17,20,21]. This is also clear from the low power needed by the human brain (approx. 20 W, where energy required to pass one spike through a synapse is 10 fJ [22]) despite large tactile data coming from receptors distributed in the skin (figure 2). Therefore, the distributed computations make the skin inherently power-efficient and robust to noise [2,3,11–14,21,23,24]. Clearly, the harmonized energy and computing in eSkin should be explored much like the distributed touch sensing has been. Another way of reducing or managing the tactile data in eSkin and attaining efficient computation is to acquire the tactile data from contact area only. There is need for development of energy transfer technologies such as new wireless protocols [8–10,25,26].

Interestingly, neuroscience studies have debated that the spiking option of data transfer in skin is not necessarily a preferred option when it comes to signal processing in the nervous system. This is because signal transfer in the nervous system is essentially aiming to maximize the fidelity with the time-continuous voltage signals (i.e. analogue signals) from receptors (or the neuron inside the brain) [12]. But the spike generation mechanism takes that signal as a probability density function and converts it into spikes (and ‘discretizes’ the signal). Hence, in the spike generation step, we inevitably lose fidelity. But, in biology, it is not possible to transmit time-continuous signals because the biological cabling system (axons) cannot support long-distance transfer of information due to electrical and communication bandwidth constraints (figure 1a) [15–19]. Hence, biology made the trade-off of losing signal quality while solving the critical signal transfer issue.

In engineered systems such as eSkin, one could probably get around this problem of signal transfer and instead transmit the time-continuous voltage signal (e.g. using single cable for each sensor). This saves a lot of computational power and processing effort compared to spike-based solutions, which require additional modules to generate spikes. However, in practice, such schemes are impractical as the large number of cables will add to the weight and complexity of eSkin. For this reason, the tactile information, in the touch-sensitive systems reported thus far, is largely transmitted serially, even if this comes at the expense of readout latency and high energy requirements. For example, the ‘always-on’ device type approach used today (figure 3a) can be extremely inefficient to handle huge tactile data from whole body, as standard digital architectures over large area eSkin will consume power—in order to run clocks, mixers and analogue/digital (A/D) converters, etc.—irrespective of the relevance of the information being recorded and processed. The above issues with current engineered systems could possibly be resolved, just like in biology, by converting the signal into spikes, before it is transferred (figure 3b). With the network of sensors as local soft and flexible computing units, and sending partially computed tactile data to the higher perception levels, the parasitic losses (particularly the CV²F) are also minimized [12,27]. The CV²F losses arises as a result of transmitting huge amounts of data to higher perception levels using complex and longer interconnects in limited communication bandwidth, which may result the loss of useful information.

Figure 3.

(a) Conventional sensor readout and transmission in a robotic hand. When the robotic hand touches an object, the analogue tactile data are converted into digital form and the train of digital pulses is transmitted to the higher perceptual levels where the entire information is processed, and action signals are transmitted back as digital pulses. This digital data are processed again and converted to analogue form to actuate the motors to obtain required movements; (b) a spike-domain sensing-processing learning-communication based on neuromorphic sensors and artificial intelligence in eSkin. (Online version in colour.)

An interesting and useful effect in soft skin arises from the integrated information from large populations of receptors that are activated during the simplest interaction with actual objects. This indicates that, when skin comes into contact with objects, the structure of skin, its morphology, micromechanical and tribology properties modulate the collective response of receptors. For example, the viscoelastic medium between contact point on the skin surface and the receptor deep inside the soft skin could alter the response of receptors. In addition, the neuroscience studies have confirmed the presence of plasticity or synaptic modulation in biological neural systems referring to the ability of synapse to weaken or strengthen the synaptic weights with respect to the tactile activity over the time [2–7]. These observations have profound implications for the development of future large area tactile skin.

From above discussion, it is clear that the major challenges for development of human skin like eSkin are related to distributed and interconnected networks of soft sensors, electronics over large areas and the distributed computing to allow efficient transfer of data while optimally using the limited bandwidth of communication channels [11–14,21,23,28]. The efficiency gap between artificial systems and their biological counterpart could be bridged through the neuromorphic mechanisms [12]. As briefly discussed above, the event-driven neuromorphic approaches demonstrated for vision and auditory may not be useful for large area eSkin [2–4,11–14]. Thus, eSkin over large area requires new integration paradigm with distributed sensors (equivalent of SAs and FAs) and distributed energy efficient non-volatile memory (NVM) devices [3,10,21,29] to store the information to allow local computation [3,10,17,21,29]. The development of large area eSkin comes with the host of technological challenges such as conformability of electronics and complexities involving multiple physical interactions rather than reflexive, pixelwise response of events [10,21]. For example, inputs from multiple sensory modalities (e.g. touch, audio, vision, etc.) give us a robust percept about the objects we contact [21,30]. The reflexive pixelwise response of events here refers to the event-driven vision, referring to independent pixels as events [31], whereas eSkin is entirely different as discussed above.

It is worth adding that the presence of eSkin on large area brings new opportunities too. For example, the large area could also be used to harvest energy (e.g. by integrating photovoltaic cells on whole body) to power the sensors and electronics [25]. In fact, the development of large area eSkin has revolutionized robotics by allowing new strategies based on simultaneous multiple contacts, particularly in the unstructured environments or where vision is blocked [2–4,11–14,24,32]. In unstructured environments, the complete world model is not available, and robots have to plan and execute the tasks in the presence of environmental uncertainties. In such cases, the contact or touch parameters measured through eSkin from different parts of body are helpful. For example, in minimal invasive surgery, an ultra-thin eSkin wrapped around the surgical instruments (e.g. laparoscopic instrument) could give tactile feedback from inside the body [3,4,10,12,33,34]. The environment inside the body is unstructured and one where visual feedback is poor. Modern surgical procedures are far more intricate than those of the past and the surgeon's knowledge and skilful hands may not guarantee the success of operation. The physical access through ultra-thin eSkin can therefore help a surgeon to feel stiffness of tissues through palpation of organs. The remote planets or satellites in space are other examples of unstructured environments where tactile feedback from the whole body of a robot can help to either avoid obstacles or to manipulate objects [29,32]. In an industrial setting or the home environment, the eSkin on the whole body of a robot will enable safe human–robot interaction.

The harmonious integration of energy is another feature which is needed for the efficient operation of the network of sensors/electronics in eSkin [10,13,25,26,35,36]. If power issue is not handled well, the eSkin could quickly drain the batteries of an autonomous robot. For example, about 8 W is needed to power about 1 K capacitive touch sensors on humanoid robot iCub [25,26]. Using these numbers, one could roughly estimate the power requirements for sensors and the read out electronics if the large area eSkin is to mimic the human skin [15–19]. The harmonious integration of power with sensor on eSkin could be addressed with innovative multifunctional devices. For example, the changes in the chemical reaction in a supercapacitor (SC) under stress/bending could be exploited to develop an energy storage device as well as to measure the applied pressure [37,38]. Likewise, the changes in resonant frequency of an antenna (RFID) as a result of stretching or temperature variations could give an indication of strain or temperature [39]. While discussing such opportunities, this paper surveys the harmonized integration of distributed computing and energy-autonomous devices for large area eSkin. Topics such as hardware approaches to handle large tactile data, local processing in neural-like fashion and methods like cost-effective printing for the fabrication of high-performance low-power electronic devices over large areas have been discussed.

The paper is organized as follows: the historical developments in the field of eSkin, neuromorphic tactile computing and the need for harmonized energy and computing are discussed in §1. Section 2 focuses on the necessity for low power and high-performance memory devices performing the simultaneous tasks of data storage and computation. Progress towards encoding of tactile data for large area eSkin and neuromorphic tactile skin is also discussed in this section. Various strategies towards the development energy autonomy to power the interconnected and distributed network of electronics, sensors, actuators and memory elements in eSkin are discussed in §3. The challenges, solutions and opportunities associated with the large area eSkin along with the requirement of distributed and harmonized energy, computing are detailed in §4. Finally, the summary of the conclusions and possible scope for future directions are presented in §5.

(a). Historical developments

Figure 4 shows the number of papers published every year in the field related to the discussion presented in this article. The data for plots in figure 4 have been taken from the Web of Science using keywords such as electronic skin, tactile skin, neuromorphic vision and neuromorphic tactile skin. In figure 4, a single data point exists for each of the keywords used. While there has been significant growth in eSkin, the efficient computation in the form of neuromorphic tactile sensing has just started to attract some interest (as shown from figure 4) [40–44].

Figure 4.

Total yearly publications in the field of electronic skin, tactile skin, neuromorphic vision and neuromorphic tactile sensing (Source: Web of Science). (Online version in colour.)

However, most of the work presented in the literature for neuromorphic tactile sensing has focused on software-based approaches instead of the development of new hardware. Contrary to this, the event-driven neuromorphic vision has gained significant attention over tactile sensing, attributing to the discussions made previously [11–14,45,46].

In context to large area eSkin with distributed and interconnected components, the control centres mainly comprise processing units (computers). The three major characteristics which need consideration for processing of data in eSkin are (i) the computational speed (or power), (ii) size of the memory to perform local computations and store the data for transmission, and (iii) cost [47]. Figure 5a shows the evolution of computers, computer power per cost (measured as millions instructions per second (MIPS) per $1000) and the acceleration in technology development over the years in comparison with the brain power equivalent [48–53]. The brain power equivalent per $1000 of computer made here is to compare the performance and the efficiency of the biological brain over computers in terms of computations, memory, communication (in limited bandwidth), etc. The technological advances which started in the year 1900 with the first room-sized electromechanical computer have brought us today to highly efficient integrated circuits, processors (operations J⁻¹) and MIPS/volume (l). At the same time, the cost of computing power has risen by 10¹⁷ times (e.g. from approx. 10⁻⁵ MIPS per $1000 in 1900 to approx. 10¹² MIPS per $1000 today) (figure 5a) [48–53]. As a result of continuous scaling of the device feature size, the size of the computers shrank to the size of personal computers (PCs) with higher MIPS per $1000 but at the cost of very high-power consumption and cost. By contrast, the brain power equivalent per $1000 of computer for human brain is 10¹⁶ [48–53].

Figure 5.

(a) Evolution of technology and comparison of power per cost. Brain power per $1000 of computer over the years [48–55]. (b) Available memory capacity (in bytes) for high-performance computing systems with respect to the computational speed FLOPS [48–55]. (Online version in colour.)

Further, figure 5b summarizes the available memory (in bytes) for high-performance processors over computational speed (in floating point operation per second (FLOPS)) [48–55]. The world's fastest computer by the American IBM Sequoia (developed in June 2012), Blue Gene/Q has computational speed more than 0.1 PETAFLOPS with approximately 10 TB of memory (figure 5b) and consumes 10 MW of power [48–55]. Whereas human brain (volume approx. 1.2 l) with more than 10 PB of memory performs 0.1–1 EXAFLOPS (figure 5b) and consumes only 20 W of power [48–55]. The current technology consumes MWs of power, generates more heat and requires cooling. Compared to humans, with 100 K neurons mm⁻³ (functional density 10¹⁴ l⁻¹), the current computers have more than 10 K transistors mm⁻³ (functional density 10¹⁰ l⁻¹). Therefore, human brain is a good example for harmonized energy and computing. The neuromorphic approach for distributed local computations and high parallelism might become the stepping-stone for future explorations in exascale machines to significantly reduce the power and cost. This is where the industry is turning to human brain and neuromorphic computing with massive parallelism for faster and efficient computation (see §2b for details). Especially in emerging applications, where the computational speed and efficiency offer a new benchmark [54] for intelligent systems.

It is clear from figure 5a that the high-performance silicon (Si)-based complementary metal oxide semiconductor (CMOS) technology has revolutionized several applications with sophisticated devices, circuits and systems [12,50,51,56]. With distributed sensing and computing, the eSkin is also expected to have high performance, and one way to achieve this is to use the CMOS-based touch sensors and electronics on ultra-thin chips [4,23,57–60]. However, it may not be economical to have an eSkin with CMOS-based devices over a large area and, in this context, investigations into new materials, devices and processing technologies are needed. The recent advent of printed electronics has showed a new direction for eSkin. With low-temperature cost-effective processing, simplified processing steps, compatibility with large-scale fabrication and reduced materials wastage, etc., the printed electronics route is interesting for large area electronics [10,36,61–69]. New processes such as contact and transfer printing of nanostructures from high-mobility materials are offering interesting avenues for obtaining devices with performance close to CMOS technology [10,12,13,27,36,61–72]. These new developments in the field of electronics could help to achieve large area high-performance eSkin and also trigger transformations in several fields where conventional CMOS-based electronics is used today.

2. Distributed memory and computing

For eSkin to help and execute touch-based interactive tasks on a real-time basis, devices to locally compute and store the tactile information are extremely important. As discussed in §1a, the presence of distributed memory in eSkin is projected to play an important role in locally processing the tactile information followed by storage [2,3,7,10–12,15–20]. To address the urgent needs related to huge amounts of tactile data and computation, the traditional von Neumann architecture has found its way to the conventional computing hardware [73–75].

The traditional von Neumann-based architecture makes use of an off-chip non-volatile flash memory (NVFM) (shown in figure 6) [73]. For energy efficient and intelligent computing, storing trillions of data and communicating with the off-chip components requires substantial energy and latency [76–79]. Today's von Neumann-based architecture therefore uses few megabytes (MBs) of static random-access memory (SRAM) as the synaptic on-chip NVFMs. However, the undesirable leakage current, long access times due to row-by-row/word-by-word (limited parallelism) operation, limit the use of SRAM [76–79]. As an alternative to SRAM, the design and fabrication of on-chip weight storage using three terminal Si nanowire floating gate (FG) NVFMs has been demonstrated [17].

Figure 6.

Comparison of von Neumann-based architectures. (Online version in colour.)

However, the viability of large area fabrication of eSkin to maintain the reliable performances for FG–NVFMs with significant downscaling the oxides (blocking and tunnelling) beyond the physical limit remains a challenge [80]. To further enhance the memory density and speed, investigation towards next-generation memory technology with new computing principles is needed to satisfy the ever-growing appetite for data and information [81]. To this end for on-chip weight storage and artificial synapses, emerging NVM (eNVM) devices including the memristors (resistive random-access memories), phase change memories, magnetoresistive random-access memory or spintronic devices and ferroelectric transistor-based RAM (Fe-RAM) have been anticipated and demonstrated (shown in figure 7) [17,73,75,79,82–89].

Figure 7.

eNVMs for neuromorphic computing: (a) memristor, (b) PC-RAMs, (c) STT-RAM and (d) Fe-RAM. (Online version in colour.)

(a). Memristors: memory and computing together

For the first time, Hickmott in 1962 [90] observed the presence of large current densities, negative resistances and hysteretic resistance switching in the current voltage (I–V) characteristics for insulators sandwiched between metal layers, forming the metal insulator metal (MIM) structure. In 1971, similar MIM structure was used by Chua [91] to conceptualize the ‘memristor (acronym for memory resistor)’ as a new element with inherent memory and valuable circuit properties. Memristors are attractive as they store the information related to past event through the last resistance (previous) state and offer potential for very-high-density integration at low cost of fabrication [89,91,92]. The experiments on memristor demonstrated fast bipolar, non-volatile (NV) switching, involving movement of charged molecular/ionic/atomic species, as observed in nanoscale devices [89,91–93]. Thus, nanoscale memristor devices demonstrate the potential to transform the NVM market, as the data are stored and processed in the same location and could lead to novel forms of computing [89,91–93]. Since then, several materials were investigated as a switching layer for memristor devices on rigid as well as flexible substrates. For instance, one- and two-dimensional layered materials and composites [87,94–103], binary oxides [78,104–106], nitrides [107,108], etc. In addition to storage, memristors with multiple analogue states are capable of actively processing information, via in-memory computing scheme [79,81,109,110]. A memristors-based hardware approach is proposed here for neuromorphic tactile skin, a comparison of various eNVM devices is given in table 1 [81,82,111–113], where these devices are compared in terms of efficient and distributed computing along with data storage. In addition to performance, these devices have been compared in terms of scalability, as this is an important parameter in context with large area distribution needed in eSkin [81,82,111–113].

Table 1.

Memory performance comparison [81,82,111–113]. T, transistor; C, capacitor; M, memory; S, selector; R, resistor; F, technology node. Suitability for neuromorphic computing: poor excellent.

In order to perform data storage and computation over large areas, such as in eSkin, a single memristor is not going to solve the problem. Owing to the excellent scalability and easy stack (three-dimensional (3D)) integration, the cross-bar array of memristors has been studied as a promising architecture for high storage density consuming low power [86,114–116]. However, with the cross-bar architecture, the passive memristor device encounters challenges such as cell-to-cell interference. This is owing to the parasitic leakage/sneak path which leads to current leakage through the interconnected low resistance state (unselected) cells and induces unnecessary power consumption and an improper read out [86,114–116].

To overcome such problems and supress the sneak currents, the use of selector or access devices in series with the memristors has been explored [105,117]. This means, for fully functional memory devices over large area eSkin, each memory cell must be coupled with a selector device such as a transistor (with switching speed like the memory cell) and in this regard one transistor–one memristor (1T–1M) structure, such as the one shown in figure 8, is a useful building block for neuromorphic computing [17]. The high-performance single crystal silicon transistor coupled with the amorphous TiO₂-based memristors, shown in figure 8, is reported to have negligible interference from the adjacent cells [118], shows repeatable and reliable resistive switching, endurance and retention properties even after multiple bending cycles.

Figure 8.

(a–h) Array of 1T–1M devices on flexible substrates along with its electrical characteristics. Reproduced with permission from [118]. Copyright © 2011 American Chemical Society. (Online version in colour.)

Another example of 1T–1M system is the CMOS-based neurons integrated with 8 × 8 array of Ti/Pt/TiO_2-X/Pt for spike time-dependent plasticity (STDP) (figure 9) [119]. In [119], the spike-based plasticity was induced into Ti/Pt/TiO_2-X/Pt memristors with the help of specific waveforms and aligned pre/post-synaptic pulses. The standalone neuromorphic system combines the programming of memristors, sensors with the custom neuron circuits applied on physical memristive devices. The efficient utilization of the 1T–1M structures for eSkin requires high performance and uniform response of flexible transistors and memory devices over large areas. In this regard, the large area device fabrication by printing approach (including conductive, semiconductor) is being explored [61,120–125].

Figure 9.

Micrograph of neuromorphic chip die and spike-based perceptron learning using Pt/TiO_2-X/Pt/Ti memristors, sensors integrated with CMOS. Adapted from [119]. Copyright © 2015 Mostafa et al. (Online version in colour.)

(b). Neuromorphic computing and eSkin

Neuromorphic computing, based on the computing models by Mead in 1990 [77] to replicate the cognitive functionality of human brain/biological synapses with high degree of parallelism, has recently come into prominence for sustainable computing technologies [109]. The major difference which attracts attention to neuromorphic computing architecture towards data centric and computation is its minimal power consumption [109]. The human brain is fantastically power-efficient and performs a host of activities including decision-making, reasoning and concurrent recognition, while a supercomputer consumes MWs [126]. Practically, by using CMOS-based switches, the actual power consumption may be higher. A neuromorphic chip designed and fabricated by IBM, which has approximately five times the number of transistors of an Intel processor, consumes approximately 65–75 mW of power [127]. The foremost challenge for the realization of wide-scale parallel neuromorphic systems lies within the development of an artificial synapse capable of emulating synapse functionality; for instance, analogue modulation along with ultralow power consumption and robust controllability [76–78,109].

Owing to its energy efficiency, spiking neural networks-based neuromorphic computing has started to attract interest for tactile skin. The active matrix of sensor-based approaches [128–131] is proposed as potential solutions for rapid and reliable processing of tactile data. In this regard, the event-driven spiking neural mechanisms (similar to the spike in the human nervous system) are relevant [17,76,132–136]. The neuromorphic sensors and computing architectures with various algorithms, including linear discriminant analysis, support vector machine, spiking neural network (SNN), extreme learning machine, K-nearest neighbours and Bayesian analysis, have attracted global attention [17,76,132–136]. As discussed previously, tactile neuromorphic systems require a different approach when compared with neuromorphic systems implemented for auditory and vision [10,30,137]. Recent studies on distributed tactile sensing suggest that the local processing of tactile data near the sensing elements (spike generation) [17,21,29], followed by transmitting/sending of a smaller amount of processed time-modulated train of data (spikes) is desired as it can overcome the issue of data deluge in tactile processing by reducing the amount of data to be transmitted. The electronic devices and neuromorphic circuits working in a manner similar to the human nervous system result in enhanced efficiencies and fault tolerance [17,21,29,138].

The human brain analogues neuromorphic tactile computing offers several advantages as discussed in §1a. As a result, a few reports on neuromorphic tactile computing have recently been presented [17,139–142]. For instance, the integrate-and-fire array transceiver (IFAT) based on the spike neural network with 90 nm CMOS consisting of a two-tier micro-pipelining scheme of 16 K parallel and addressable neurons (total 65 K neurons), consuming an average of 22 pJ per synaptic input event and 25 µW standby power, is demonstrated in [140]. In another work, analogues to IFAT, the field programmable gate arrays-based generic multilayer neuromorphic systems with a set of parallel neural processing elements are presented [142]. The work in [142] resulted in 12 times increased speed, 45 times reduction in energy consumption along with marginal accuracy loss comparing with CPU-only computation. Likewise, the field programmable analogue arrays (FPAA) architecture (analogues to the FG-NVFM) in the mixed mode, computational configurable analogue blocks (CAB), as shown in figure 10, was proposed in [139]. The programmable FG transistors provide computational elements and switching logic, resulting in a compact, expandable switch matrix design as well as a functionally rich CABs. Programming of individual FG in FPAA works by tuning the drain and gate voltage (similar to the NVFMs), shown in figure 10. The programmable FG transistors used in the work function as computational element and switching logic.

Figure 10.

Array of single floating gate transistors which can be selectively programmed by applying the gate and drain voltages of the columns and rows. Reproduced with permission from [139]. Copyright © 2002, Springer-Verlag Berlin Heidelberg. (Online version in colour.)

The reported neuromorphic techniques offer several advantages, but, currently, the technological challenges limit their potential usage in distributed and large area neuromorphic tactile sensing as large-scale neuromorphic circuits and systems are difficult. In this context, the printed circuits based on neural nanowire field effect transistors (υ-NWFET) are an attractive approach [17]. The illustration for the biological neuron along with the schematic illustrations of Si nanowires (NW)-based υ-NWFET devices is shown in figure 11a–c. The υ-NWFET device uses Si NWs as the active material and they can be printed over a large area of flexible substrates which allows us to carry out local/mobile data processing like biological skin. The printing of NWs on large areas has been demonstrated in past by using transfer printing or contact printing approaches [63,64,68,122,143]. The structure of the υ-NWFET devices along with the cross-sectional view demonstrating different regions (FG, Source, Drain, Gate) is shown in figure 11e. The capacitive coupling between the Si NW FG and the top gate is used in these devices to modulate the current in the channel.

Figure 11.

υ-NWFET device. (a) Schematic for biological neuron, (b) artificial neuron with corresponding weights, (c) υ-NWFET with various gate geometries, (d) schematic for the artificial hand covered with array of υ-NWFET, (e) 3D illustration of the υ-NWFET device and its equivalent circuit model. (f) Illustration of eSkin with 6 × 6 array of υ-NWFET-based HNN approach for tactile neuromorphic computing. Adapted from [17]. Copyright © 2017 Taube et al. 2017. (Online version in colour.)

The overlapping area between the individual top gate and the FG is used in these devices to obtain an equivalent of synaptic weight of the neural input (figure 11c) and the combination of gates are activated to imitate the synaptic summation of weighted inputs in the cell body (soma) of the biological neuron or the artificial neuron. The schematic illustration of eSkin with 6 × 6 array of υ-NWFET-based hardware neural network approach for neuromorphic computing is shown in figure 11d,f. The activation function performed at circuit level is discussed [17]. By extending this approach to obtain one transistor–one memristor (1T–1M) structure operating at low power, and scaling up these building blocks to obtain large area eSkin, it will be possible to develop brain-inspired computational eSkin [75,116,144–146].

The 3D printed organic device and circuits (e.g. resistive pressure sensor, organic ring oscillator and the synaptic transistor)-based flexible sensory system mimicking the functions of a sensory nerve of human skin to measure the applied force is among one of the few examples in this direction (as shown in figure 12) [141]. Here, the ring oscillator converts the data from multiple tactile receptors (resistive pressure sensors) to voltage pulses, after which the signals are converted into post-synaptic currents with the help of a synaptic transistor. By transmitting the entire set of information, the developed system still requires handling the data deluge in the limited band width. From the neuromorphic tactile implementations reported so far, it turns out that the focus remains on emulating either FAs or SAs mechanoreceptors to spiking activity and physical transduction properties (for example, piezoelectric corresponding to FAs; resistive and capacitive corresponding to SAs). The future challenges include the development of a hybrid system with transduction sites of dynamic range, increased sensitivity and mechanical characteristics, better resolution and varying bandwidth to successfully demonstrate neural models which can take into account the different adaptation characteristics of the receptors in the skin.

Figure 12.

Bioinspired flexible organic devices and circuits. Adapted from [141]. Copyright © 2018, Science. (Online version in colour.)

In this context, our recent work demonstrates detection of both static and dynamic stimuli in eSkin by using a stack of piezoelectric (for dynamic stimuli) and capacitive (for static or quasi-static stimuli) sensors [4]. By mimicking the structures and functions of the skin in human fingertips, we have presented a highly sensitive capacitive-piezoelectric flexible sensing skin with fingerprint-like patterns to detect and discriminate between spatio-temporal tactile stimuli including static and dynamic pressures and textures. The intrinsic inability of the piezoelectric sensors to detect sustained static pressures is overcome by using the integrated capacitive sensor. Thus, the capacitive-piezoelectric sensor stack could mimic the behaviour of SA and FA mechanoreceptors in the human skin. The dynamic tactile-sensing element (i.e. piezoelectric sensing part of the stack) was interfaced with n-metal oxide semiconductor field effect transistor [58] in the extended gate mode [44,60,147,148]. This arrangement is similar to our piezoelectric oxide semiconductor field effect transistor devices reported earlier [149–152]. Likewise, the static capacitive sensor's output was converted into an equivalent digital signal using a charge time measure unit. The analogue outputs from the sensing stack were digitized and sent to a PC.

As a proof of concept, we showed the neural-like processing tactile data from the sensing stack to mimic a real-world scenario where textures are distinguished by touching planar and non-planar surfaces. To this end, a biologically plausible wavelet transform was used to encode the sensor stack output into spike trains based on the leaky integrate-and-fire (LIF) model. Spike trains were then classified with a tempotron classifier, using a biologically observed STDP mechanism-based learning rule. Since the spiking trains in this work were obtained after sending all digital data to the PC, the process is not entirely similar to the way the spikes are generated by the receptors in the skin. For example, the transferring of all the tactile data to the PC does not align fully with the discussion above about the skin opting for spiking activity due to communication bandwidth limitations. Such issues could be overcome in the future with hardware neural network (HNN) implementation using devices such as neural nanowire FET (υ-NWFET), explained earlier. For example, multiple gates of υ-NWFET could be connected to multiple touch sensors (dynamic or static) which together could modulate the device response—generating spike only if the combined output is more than predefined value. Otherwise, υ-NWFET remains in the OFF state. The predefined value could be set by a memristor connected with υ-NWFET, described previously. Further, like biological solutions, a distributed local processing of tactile data with partial processing close to the sensing elements and sending of smaller amounts of processed data to higher perceptual levels could be advantageous as an engineered solution as well. Finally, it will be of great importance if the final neuromorphic systems include the sensing of temperature, pain and proprioception. The summary of the neuromorphic computing platforms for eSkin is presented in table 2.

Table 2.

Examples of neural-like computing devices for eSkin.

approach	computation/ memory devices	neuron type	hardware/ software	synaptic weight/ plasticity	ref.
STDP	memristor	analogue	both	STDP	[119]
IFAT	—	analogue	software	IFAT	[140]
FPAA	analogue blocks, FG-NVFM	analogue	software	—	[139]
neural nanowire field effect transistors	FG-NVFM	hardware neural network	both	overlapping area between individual gates	[17]
artificial afferent nerve/SNN	synaptic transistor	SNN	hardware	—	[141]

3. Energy autonomy in eSkin

Apart from the high energy requirements, light weight and conformability of energy devices are major challenges for eSkin. The batteries used as a source of energy to fuel the sensors and associated electronics are generally heavy/non-conformable and require frequent re-charging. Such bottlenecks can be overcome by using alternative skin conformable energy storage devices coupled to the energy harvesters [25,26]. Some solutions that have been reported in recent years for self-powered wearable health monitoring patches are noteworthy. These include wireless-powered and solar-powered sensing patches with efficient low-power electronics [10,25,26,37,153]. The use of these low-power sensors based on materials such as indium tin oxide (ITO) (approx. 100 µW cm⁻²) [17] and graphene (approx. 20 nW cm⁻²) [25,26] is another alternative to alleviate the issue to some extent. For example, with just approximately 20 nW needed per cm², the graphene-based sensors require 3.9 µW over an area of 1.5 m² (as shown in figure 13) [25,26]. Even if these are simplistic calculations (e.g. the power requirement for electronics to read out and process the sensors data is not included), these examples indicate that the energy autonomy of eSkin has started to receive attention. This is also evident from recent review articles where various solutions for energy-autonomous eSkin have been discussed [25,26] along with components such as energy generator, storage, self-powered systems and various applications.

Figure 13.

(a) Three-dimensional schematic for graphene-based self-powered eSkin. Inset in (a) shows the photograph of the flexible sensor. (b) Photograph of flexible graphene capacitive touch sensor. (c) Flexible graphene-based touch sensor integrated on to a robotic hand. Self-powered eSkin used on the robotic hand with tactile feedback: (d) OFF and (e) ON. Reproduced with permission from Núñez et al. [25,26]. Copyright © 2019 Springer Nature and Copyright © 2017 Published by WILEY-VCH Verlag Gm. (Online version in colour.)

As discussed in §1a, the presence of skin on a large area offers an excellent opportunity to harvest energy from the ambient. For example, by using conformable solar cells over the whole body, it is possible to generate enough power to operate the eSkin [25,26,154]. With a wide range of power density (10–500 W m⁻²) [25,26] made possible with photovoltaic (PV) cells, it would be feasible to operate various sensors, actuators and other electronic components on the robotic or prosthetic hand. In one of our previous works, we integrated the graphene-based coplanar capacitive touch sensors on PV cells to obtain self-powered tactile eSkin [25,26]. The key feature of the graphene-based eSkin is the transparency (sunlight absorption below 3%), which allows most of the light to reach the PV cell underneath to harvest energy. The graphene-based touch-sensitive layer (sensitivity 4.3 kPa⁻¹ over 0.11–80 kPa range) requires ultralow power (approx. 20 nW cm⁻²), which is much lower than the power generated by the flexible a-Si-based PV cells (output power density of approx. 19 mW cm⁻²). The surplus energy could be stored for later use (e.g. when there is no sunlight) by integrating with this eSkin a flexible energy storage device to obtain a self-powered eSkin which can continuously operate. The stored energy could also be used to operate the actuators in robots or prosthetic hand as recently demonstrated with flexible SCs integrated underneath the solar cells [25,26]. It was demonstrated that the solar-powered energy packs, constituting a solar cell and flexible SC, can actuate the robot hand. The output voltage of these SCs (approx. 2.25 V) is compatible with the requirements of most of the electronics in use today. Adding wireless power transfer technology [155] to these SCs and energy generators will add significant value in terms of power management.

The amount of generated energy will be significant if we consider the presence of touch sensors over an area equivalent to an adult human skin (approx. 2 m²). Thus, the presence of eSkin over a large area has also its distinct advantage. The demonstrations such as the one described above are good examples of turning challenges (e.g. large area of eSkin has often been cited as bottleneck) into opportunities. The self-powering capabilities including solar and wireless-powered energy devices have also been demonstrated for a small patch of sensors for wearable health monitoring [25,26,37]. With the growing requirements for soft eSkin, it is important to investigate the integration of alternative clean and portable energy harvesting approaches, for example piezoelectric, thermoelectric, triboelectric and electromagnetic mechanisms, etc. The thermoelectric energy harvesters coupled to the energy storage devices could tap the heat generated by the actuators on robots and fuel the energy storage devices for powering the electronics in the eSkin. Similarly, the piezoelectric harvesters can be used to generate electricity as a result of multiple physical interactions or mechanical vibrations in the environment and use the same to fuel the low-power electronics in the eSkin. Likewise, the triboelectric nanogenerators (TENGs) working on the principle of energy generation under mechanical deformations such as pressing, touching, bending and stretching are most interesting for eSkin applications and have recently gained significant attention [38,156–158]. In fact, a TENG based on single electrode with a load of 0.1 GΩ has been recently shown to deliver a power density of 500 mW m⁻² by harvesting biomechanical energy for self-powered tactile sensor applications [156]. The skin conformable TENG can be coupled to the energy storage devices such as the micro-SC for energy-autonomous self-powered sensor applications [157]. The summary of various self-powered energy devices for large area flexible eSkin is presented in table 3.

Table 3.

Examples of self-powered energy devices for flexible eSkin.

energy technology	material	energy storage	power density (mW cm−12)	ref.
solid-state PV cells	Si	flexible SCs	∼19	[26]
SCs	graphene foam	flexible SCs	∼0.27	[37]
SCs	graphene polyuretahne composite	flexible SCs	11.15	[38]
TENG	modified polydimethylsiloxane (PDMS)	—	500	[156]
TENG	graphene	flexible µSCs	800	[157]
TENG	PDMS, neoprene rubber, paper	—	—	[158]

4. Challenges and opportunities

As clear from figure 4, very few reports are present in the literature regarding neuromorphic tactile sensing. Most of the work demonstrated so far comprises single or a small array of memristors, while large-scale integration of memristor in the context of eSkin is not there yet. This lack of advances in neuromorphic tactile sensing is due to the challenges related to the integration of a large number of memristors into the cross-bar matrix over large areas with minimal device to device variation etc. Further, the requirement of local computation to effectively use the available bandwidth and to minimize the parasitic will be intensified with large area. This potentially raises another concern in the way of obtaining precision of the weighted sum of the array. Moreover, the energy management to effectively drive the distributed array of memristors, sensors/electronics to perform the computation and provide the real-time feedback requires attention too.

A network of interconnected high-performance devices over large areas on flexible substrates is one of the main requirements for next-generation eSkin, along with emerging applications such as wearables and IoT. Owing to the limitations of current fabrication technologies and realization of flexible devices distributed over large areas for eSkin, new alternative and cost-effective approaches such as printing (inkjet, contact, stamp, transfer, roll to roll) of functional layers (metals, semiconductors, insulators) are being explored [20,46,59,68,128,159–163]. Scaling down the size of the memristor devices is reported as one of the possible solutions to minimize the device-to-device variation for large area eSkin applications. The variability in memristor devices is generally associated with the randomness of the filament formation process. With a single filament and volume comparable to that of the device, the uniformity of the devices can be improved. Reducing the size of memristor devices may also help to lower the programming current and as a result leads to reduced power consumption. The state-of-the-art memristor feature size reported is around 10 nm, while in theory, it can be scaled to the atomic level [54,73,78,79,81,104,109,134,164]. In addition, the cognitive operations in the high-dimensional space rely on large amounts of weights, thus requiring high density of memristor. Hence, the 3D vertical stacking of memristor crossbars can be advantageous in terms of integration density per area [114]. The 3D stacking of memristor could also significantly reduce the issue of parasitic resistances for individual cross-bar layer [114]. Innovative 3D stacking, for example, the self-rectifying memristors with high device packing density [165], offer interesting route with no requirement for external selector devices to suppress the intra/inter-layer sneak path currents. These examples illustrate the opportunity for low-cost and high-density integration of memristors and associated electronics for neuromorphic tactile sensing.

With the help of various approaches discussed previously (in §3), once the energy required to power the sensors and distributed electronics in the eSkin is available, the efficient utilization and distributed management of available energy to fuel the electronics in the eSkin remains a challenge. For instance, if the robotic hand performs a simple task of pick and place, the mechanoreceptors in contact/touching the object should only be active and perform local computing, etc. (Figure 1a). With proper management and distribution of power in the eSkin, a considerable amount of energy can be saved. In this context, the recent developments related to the use of artificial intelligent (AI) for efficient distribution and operation for zero power IoT are noteworthy [166,167]. There is potential for using similar approaches for power management in eSkin, for example to direct the energy to areas requiring greater attention instead of powering the whole skin or to identify the skin areas that are frequently used. The AI techniques could also advance the energy-autonomous skin. For example, we recently reported energy-autonomous eSkin-based graphene-based transparent touch sensors integrated on solar cells [25–27]. The AI methods could be used in such eSkin configurations to dynamically change of the orientation of robotic limbs to allow maximum exposure to light and hence to generate more power. The other techniques for energy harvesting or efficient energy management include optimized power systems using the latest technologies, for instance battery-less operations and wireless power transfer. Compared with wireless power transfer using Zigbee or Bluetooth enabled with sensor node generally consuming 25–30 mW of power, the designs for energy efficient system-on-chips consuming 2–4 mW [168] of active power could be advantageous for energy-aware eSkin. Although these approaches are in the early stage, they could make good contributions to the set of solutions needed to address the challenges related to energy or power management in eSkin.

5. Conclusion and discussions

In humans, tactile perception is the most important source to plan and execute tasks involving multiple physical interaction. Likewise, the eSkin in robots mimics human skin to provide tactile feedback for safe interaction with the environment. Despite significant advancements, several challenges associated with the development of eSkin still persist apart from realization of large area eSkin with distributed computing and harmonized energy. An overview of alternative printing technologies and materials for development of large area eSkin and computing platforms using eNVM (especially memristor) for efficient computing and data storage is presented here. The harmonious integration of distributed computing and energy on large area eSkin discussed in this paper is surely an area which requires immediate attention. To cope up with the shortcomings of the conventional computing systems, modified von Neumann architecture-based computing platforms using emerging NV memories (especially memristor) have been discussed here for efficient computing and handling of tactile data. The computing approaches similar to the human brain (neuromorphic computing) have been employed in the past for auditory and vision with centralized computing in smaller area. Whereas large area eSkin requires a paradigm shift performing distributed computing. The paper also focused on challenges from the point of view of hardware development. These include large area fabrication of high-performance devices (e.g. printed sensors and electronics) consuming low power and 3D integration. The possibility to scale down the dimension of the memristors, integrate as a 3D stack, and its importance to minimize the parasitic, device variability, etc. highlighted here are projected to tackle some of the challenges, although some additional issues may possibly emerge, which could demand for additional efforts. While the focus of the paper is on large area eSkin, the discussion presented here also applies to applications where network of multifunctional sensors are needed over large areas. These include intelligent health monitoring, wearable systems, IoT and consumer electronics.

Data accessibility

This article does not contain any additional data.

Authors' contributions

R.D. proposed the idea behind this invited paper. M.S. and R.D. wrote the manuscript. RD provided overall supervision.

Competing interests

We declare we have no competing interests.

Funding

This work was supported EPSRC Engineering Fellowship for Growth-PRINTSKIN (grant no. EP/M002527/1) and neuPRINTSKIN (grant no. EP/R029644/1).