Flexible artificial Si-In-Zn-O/ion gel synapse and its application to sensory-neuromorphic system for sign language translation

Flexible artificial synapses are fabricated with a SIZO/ion gel hybrid structure forming a polyimide substrate.

Abstract

We propose a flexible artificial synapse based on a silicon-indium-zinc-oxide (SIZO)/ion gel hybrid structure directly fabricated on a polyimide substrate, where the channel conductance is effectively modulated via ion movement in the ion gel. This synaptic operation is possible because of the low-temperature deposition process of the SIZO layer (<150°C) and the surface roughness improvement of the poly(4-vinylphenol) buffer layer (12.29→1.81 nm). The flexible synaptic device exhibits extremely stable synaptic performance under high mechanical (bending 1500 times with a radius of 5 mm) and electrical stress (application of voltage pulses 10⁴ times) without any degradation. Last, a sensory-neuromorphic system for sign language translation, which consists of stretchable resistive sensors and flexible artificial synapses, is designed and successfully evaluated via training and recognition simulation using hand sign patterns obtained by stretchable sensors (maximum recognition rate, 99.4%).

INTRODUCTION

Because Internet of Things (IoT) technologies are being used by not only industries but also individuals, there has been a rapid increase in the demand for portable smart electronic devices that can be connected to the data cloud networks anywhere and anytime (1–3). In particular, flexible electronics have recently attracted notable attention as future technologies related to robotics and smart health care systems because of their universal applicability and mechanical flexibility (4–7). In recent years, there have been many efforts to implement flexible electronic devices using various materials, such as organic thin films (8–10), two-dimensional (2D) materials (11–13), and amorphous oxide semiconductors (AOSs) (14–16). Among them, AOSs have shown good potential for flexible electronics in terms of fabrication process, such as low cost, high reliability, and low-temperature processing (17, 18). Choi et al. (19) reported a flexible silicon-indium-zinc-oxide (SIZO) thin-film transistor (TFT) with good mechanical stability at high strain. The SIZO-TFT was fabricated on a flexible polyimide (PI) substrate at 150°C. Sim et al. (20) demonstrated multifunctional stretchable human-machine interface (HMI) devices using sol-gel-on-polymer-processed IZO semiconductor nanomembranes. The HMI devices can be easily worn by humans and can also be implemented as prosthetic skin for robotics. Cantatore and co-workers (21) implemented a flexible electrocardiogram (ECG) patch on foil with self-aligned indium-gallium-zinc-oxide TFTs. The flexible ECG patch could amplify the measured analog ECG signals and convert them to a digital output for biomedical analysis.

Meanwhile, the demand for future artificial intelligence systems has led to an expansion in the research scope of flexible electronics to neuromorphic devices. Most of the relevant research has focused on the development of synaptic devices, which are the most important components of artificial neural networks (ANNs) for neuromorphic computing technology. Recently, various flexible synaptic devices have been reported (22–25). Shim et al. (26) reported a stretchable synaptic transistor based on elastomeric electronic materials, where its synaptic characteristics were retained even when stretched to 150%. Ham et al. (27) fabricated a 1D fiber-shaped wearable synapse based on a ferroelectric organic transistor, which exhibited excellent reliability under 6000 repeated input stimuli and mechanical bending stress. Zhao et al. (28) implemented an ionic polymer–based flexible synapse, where its synaptic performance was sustained even when bent to 360° and under 10⁴ pulse cycles.

Interesting studies on flexible synaptic devices have been recently reported, where a simple neural system consisting of artificial synapses and sensors was demonstrated (29–33). This neural system was referred to as a sensory-neuromorphic system; the system mimicked the nervous system, was capable of the transduction and processing of signals sensed in real time, and afforded biological benefits such as fault tolerance and low power consumption. Thus far, many studies on sensory-neuromorphic systems have been conducted. For instance, in 2017, a tactile perception system consisting of organic-based synaptic transistors and pressure sensors was proposed by Zang et al. (31). Kim et al. (32), in 2018, implemented an artificial afferent nerve system composed of flexible synaptic transistors and pressure sensors, highlighting that the synaptic transistor can be used to interface with biological efferent nerves to form a complete monosynaptic reflex arc. In 2020, Kim et al. (33) reported a tactile sensory-neuromorphic system based on tactile sensors and carbon nanotube (CNT)–based synapses. They highlighted the potential to further improve the technologies used in cyborg and neuromorphic system by endowing supramodal perceptual capabilities. Although the proof of feasibility of the flexible synaptic devices and sensor-embedded synapses was performed, more studies on synaptic performance evaluation are still required for the application to complex ANNs. Moreover, there have been few studies involving an entire system-level performance investigation via the training and recognition tasks using data obtained by the sensors thoroughly.

Here, we report a flexible synaptic device based on a SIZO/ion gel hybrid structure that maintains a highly stable performance under mechanical and electrical stress. The SIZO semiconducting channel was successfully formed on a poly(4-vinylphenol) (PVP)–coated flexible PI substrate via the low-temperature SIZO deposition process, and the surface roughness was improved by the PVP buffer layer. To evaluate the performance stability with mechanical flexibility and durability, we investigate the static and dynamic characteristics of the synaptic device with respect to the bending radius and cycle. Then, the optimization of the dynamic range (G_max/G_min) and asymmetricity (AS) of long-term potentiation/depression (LTP/D) is conducted via pulse engineering, followed by an examination of the relevant cycle-to-cycle variations. Last, we conduct training and recognition tasks for sign language patterns obtained from prefabricated stretchable sensors and evaluate the performance of flexible sensory-neuromorphic systems.

RESULTS

Flexible synaptic device based on the SIZO/ion gel hybrid structure

A biological synapse is located at the connection between two adjacent neurons, where the strength of the connected neural pathways is called the “synaptic weight.” The synaptic weight is determined by the storage of information, and it is strengthened or weakened by the release of neurotransmitters (34). To mimic the configuration and functions of the biological synapse, we fabricated an artificial synapse with a SIZO/ion gel hybrid structure, as shown in the schematic images in Fig. 1A. Here, the SIZO semiconducting channel is located between two Au electrodes, which denote presynaptic (T_pre) and postsynaptic terminals (T_post), respectively. When a presynaptic voltage (V_pre) is applied to T_pre, a postsynaptic current (I_post) flows from T_pre to T_post; the postsynaptic conductance (G_post) is determined using the following equation: G_post = I_post/V_pre. Moreover, G_post is modulated by the movement of mobile ions in the ion gel via the weight control voltage (V_WC). Figure 1B shows the flexible synaptic device array with a size of 3 × 3, which was fabricated directly on the PVP-coated PI substrate, and the optical microscopy (OM) image of the magnified single device. This was possible because of the low-temperature deposition process of the SIZO channel layer (<150°C) and the minimized surface roughness by the PVP buffer layer. We introduced PVP buffer layers of 1 weight % (wt %) and 3 wt % between the SIZO channel and the PI substrate, followed by an investigation of the surface roughness via atomic force microscopy (AFM). As seen in the AFM images in Fig. 1C, the surface roughness was slightly improved after introducing the PVP buffer layer of 1 wt % and further improved with a 3 wt % PVP layer. We calculated the root mean square of the microscopic peaks and valleys measured at the surfaces (RMS_roughness), as shown in Fig. 1D. The RMS_roughness value was improved significantly from 12.29 to 1.81 nm, which clearly indicates that the PVP buffer layer significantly contributed to the direct deposition of the SIZO channel layer on the PI substrate. This improvement in the surface roughness is attributed to the high viscosity of the PVP solution (35). We then investigated the static postsynaptic current versus weight control voltage (I_post-V_WC) characteristics of the synaptic device (Fig. 1E). Compared with the device without the PVP layer, the 3 wt % PVP–inserted device exhibited a I_post-V_WC characteristic curve with relatively high on current, high on/off current ratio, and large memory window, which may be due to the improvement in the surface roughness. The decreased surface roughness improves the quality of the PVP-SIZO interface, which, in turn, suppresses interfacial scattering and trapping. This is expected to improve the I_post-V_WC characteristics. For this measurement, we swept the V_WC from −20 to +20 V (first sweep: forward direction) and from +20 to −20 V (second sweep: reverse direction) and applied a constant V_pre of 1 V. The memory window represents the difference between the threshold voltage (V_th) values of the first and second curves (ΔV_th).

Fig. 1. Flexible synaptic device based on the SIZO/ion gel hybrid structure.

(A) Schematic images of the flexible synaptic device based on a SIZO/ion gel hybrid structure. (B) Flexible synaptic device array with a size of 3 × 3, which was fabricated directly on the PVP-coated PI substrate and the OM image of the magnified single device. (C) AFM mapping images of the SIZO surface without/with the insertion of the PVP buffer layer. (D) Calculated RMS_roughness values with respect to the PVP concentration. (E) Static I_post-V_WC characteristic of the synaptic device without PVP buffer layer (black) and 3 wt % PVP buffer layer (wine). (F) Conductance response of the flexible synaptic device according to the continuous V_WC pulses, where the pulse width and frequency were fixed as 50 ms and 5 Hz, respectively. (G and H) Weight update mechanism of the SIZO/ion gel synaptic device when the positive (G) and negative V_WC (H) was applied, respectively. The left panels are illustrations of the ion movement in the SIZO/ion gel synaptic device, and the right panels are the energy band diagrams according to the applied V_WC (photo credit: Jeong-Ick Cho, Sungkyunkwan University).

We then investigated the conductance response of the flexible synaptic device according to the continuous V_WC pulses (Fig. 1F), where the pulse width and frequency were fixed at 50 ms and 5 Hz, respectively. To monitor the conductance change, a V_pre value of 1 V was applied to T_pre. Because 100 positive V_WC pulses of +5 V were applied to the weight control terminal (T_WC), G_post increased linearly from 0.6 to 27.8 nS. It also decreased gradually to the initial conductance level after applying 100 negative V_WC pulses of −5 V. This result is quite impressive because an artificial synapse should have a multistate that can be modulated linearly and reproducibly for the successful implementation of a hardware neural network (HW-NN) (36–41). This conductance response originates from the gradual ion movement inside the ion gel according to V_WC (Fig. 1G and fig. S1). When a positive V_WC is applied, the positive ions ([EMIM]⁺) in the ion gel are pushed away from T_WC to the SIZO channel, and the negative ions ([TFSI]⁻) are simultaneously pulled to T_WC (see SIZO/ion gel interface). As indicated by the energy band diagrams in Fig. 1G, the positive ions induce the accumulation of electron carriers in the SIZO channel region, thereby increasing the surface potential. This increases the electron injection probability at the postsynaptic terminal/SIZO junction, leading to the conductance potentiation phenomenon. On the other hand, when a negative V_WC is applied, the positive ions are dragged to T_WC, and the negative ions move to the SIZO channel region at the same time. Similarly, as indicated by the energy band diagrams in Fig. 1H, the negative ions induce the accumulation of hole carriers in the SIZO channel, leading to a decrease in the surface potential. Thus, the probability of electron injection at the presynaptic terminal/SIZO junction decreases, leading to the conductance depression phenomenon (see also fig. S1). The ions in the ion gel move toward the SIZO surface region because of the positive voltage pulses and form a temporary electric double layer (EDL) at the interface between the ion gel and the SIZO (42, 43). When applying the negative voltage pulses, the accumulated ions diffuse back into the ion gel medium, gradually decomposing the EDL (43). In the case of the proposed device, when a positive V_WC is applied (i.e., V_WC > 5 V), the ions accumulate near the interface and are also adsorbed into the existing defects in the SIZO channel region. The ions that form the EDL rapidly diffuse into the ion gel under a negative V_WC; however, the adsorbed ions are expected to be removed slowly, which likely causes the short-term and long-term synaptic plasticity (see also fig. S2 for short-term synaptic plasticity).

Mechanical flexibility and durability of SIZO/ion gel synaptic device

In flexible neuromorphic electronics, it is important to maintain stable electrical characteristics under mechanical stresses that occur when the synaptic device is stretched or bent. In particular, robustness against tensile stress by bending is a key factor for the application of synaptic devices in future flexible electronics such as wearable computing, soft robotics, and neuroprosthetics (44). In this section, to demonstrate the mechanical flexibility and durability of the SIZO/ion gel–based artificial synapse, we investigated the synaptic performance stability with respect to the bending radius and cycle. For the bending flexibility test, a synaptic device array fabricated on a PI film with a size of 2 × 2 cm² was attached to the arch-shaped metal bars with radii of 30, 10, or 5 mm using scotch tape, as shown in Fig. 2A and fig. S3A. The static I_post-V_WC characteristics were then analyzed with respect to the bending radius (Fig. 2B). The V_WC was swept from −20 to +20 V (first sweep: forward direction) and from +20 to −20 V (second sweep: reverse direction), where a constant V_pre of 1 V was applied to T_pre. In all the bending cases, the I_post-V_WC characteristics were not changed significantly. As shown in the top panel of Fig. 2D, ΔV_th barely changed at approximately 20 V, even when bent to a radius of 5 mm. The relative standard deviation (RSD) was only 1.1%, indicating excellent stability for mechanical flexibility (RSD = SD/mean). We then investigated the dynamic characteristics of synaptic devices with respect to the bending radius. For each radius, the response of G_post to 100 positive and 100 negative V_WC voltage pulses was monitored, where the magnitude, width, and frequency of the pulse were 5 V, 50 ms, and 5 Hz, respectively. As shown in Fig. 2C, the measured LTP/D characteristic curves were very similar to each other in all bending radius tests. Although the extracted ΔG_post (= G_max − G_min) slightly changed from 13.5 to 11.25 nS as the device was bent to 5 mm, the RSD for the bending radius was still as small as 8.7% (bottom panel of Fig. 2D). The stable device flexibility is attributed to (i) the excellent flexibility of the SIZO (19, 45) and ion gel (46, 47) layers and (ii) the extremely small bending strain (ε) of 0.26%, originating from the significantly low device thickness (fig. S4).

Fig. 2. Mechanical flexibility and durability of the SIZO/ion gel synaptic device.

(A) Measurement images for mechanical flexibility of the SIZO/ion gel synaptic device. The flexible synapse was attached to the arch-shaped metal bars with a radius of 30, 10, or 5 mm using scotch tape. (B and C) Static I_post-V_WC and dynamic LTP/D characteristics with respect to the bending radius. (D) Extracted ΔV_th (top panel) and ΔG_post (bottom panel) and their RSDs with respect to the bending radius. (E) Measurement images for mechanical durability of the SIZO/ion gel synaptic device. The side parts of the PI film including the synaptic device array were attached to the bending machine using scotch tape, and then, we operated the bending machine to apply tensile stress to the device repeatedly 500, 1000, and 1500 times. (F and G) Static I_post-V_WC and dynamic LTP/D characteristics of the synaptic device with regard to the bending cycle. (H) Extracted ΔV_th (top panel) and ΔG_post (bottom panel) and their RSDs with regard to the bending cycle (photo credit: Jeong-Ick Cho, Sungkyunkwan University).

We also examined the mechanical durability of the bending cycle with the SIZO/ion gel artificial synapse, where the bending radius was set to 5 mm and the bending speed was controlled using a commercial bending machine (ECOPIA, step motor controller, SMC 100). As shown in Fig. 2E and fig. S3B, the side parts of the PI film, including the synaptic device array, were attached to the bending machine using scotch tape, and the bending machine then repeatedly applied tensile stress to the device. After 500, 1000, and 1500 repeated bending cycles, we placed the device sample onto the flat substrate and analyzed the I_post-V_WC and LTP/D characteristics, as shown in Fig. 2 (B and C). In the static characteristic analysis, the synaptic device exhibited very stable I_post-V_WC characteristics with regard to the bending cycle; the V_th values of the first and second curves were maintained even after 1500 bending cycles (Fig. 2F). The calculated ΔV_th values were constant at approximately 20 V, and their RSD was only 1.4%, indicating that the synaptic device had excellent durability against the bending cycle (upper panel of Fig. 2H). Moreover, the device exhibited very stable LTP/D characteristics, where the conductance modulation by the voltage pulses was perfectly repeated even for 1500 bending cycles (Fig. 2G). The extracted ΔG_post values showed little fluctuation, but their RSD was only as small as 2.9% (bottom panel of Fig. 2H). In addition, we conducted a performance benchmark in comparison with other studies on the AOS-based flexible synapses (table S1). Our device showed relatively high flexibility (bending radius of 5 mm) and high durability (bending cycle of 1.5 × 10³).

Controllability and electrical stability of dynamic characteristics

The performance of an HW-NN is strongly dependent on the dynamic characteristics of the synaptic devices composing the network. In particular, G_max/G_min and AS of the LTP/D characteristics are the most important factors that affect the learning accuracy of the HW-NN (48). Thus, it is important to investigate the controllability of synaptic weight modulation and to determine the optimized condition of the weight control signal (V_WC in this study) for achieving high G_max/G_min and low AS for a high learning accuracy. To evaluate the weight modulation ability of the device and investigate the optimal pulse condition, we monitored G_post by applying to T_WC 100 potentiating and 100 depressing pulses with various pulse amplitude and width conditions (Fig. 3A). We applied V_WC pulses, whose potentiating and depressing pulse amplitudes varied from ±3 to ±5 V (step no. 1), where the pulse width and frequency were set to 40 ms and 5 Hz, respectively. To visualize the weight modulation trends in the LTP/D characteristic curves, we normalized G_post ranging from 0 to 1 (top panel of Fig. 3B). As the pulse amplitude increased from ±3 to ±5 V, the LTP region was maintained, but G_post in the LTD region decreased rapidly. To investigate the conductance modulation quantitatively, we extracted G_max/G_min and AS from the LTP/D characteristic curves (bottom panel of Fig. 3B). Here, G_max/G_min is the ratio of the maximum and minimum conductance values, and AS is the difference between the nonlinearities in the LTP and LTD regions (NL_P − NL_D) (see the detailed nonlinearity extraction method in fig. S5). When a V_WC value of ±3 V was applied to the device, G_max/G_min was too low (4.3), which was not sufficient for the minimum requirement of the dynamic range (>10) for learning. As the pulse amplitude was increased to ±5 V, G_max/G_min also increased to 11.0, which was slightly higher than the minimum requirement, but it was still too low to achieve a high learning accuracy. Meanwhile, AS degraded from 4.62 to 4.95 as the pulse amplitude increased from ±3 to ±5 V. This result indicates that G_max/G_min and AS have a trade-off relationship and can be controlled by adjusting the pulse energy applied to T_WC. To further improve G_max/G_min, we adjusted the pulse width of V_WC (step no. 2) while maintaining the pulse amplitude and frequency at 5 and 5 Hz, respectively (Fig. 3C). Because the pulse width was varied from 40 to 100 ms, the LTP/D characteristic became more nonlinear, and the characteristic curve in the LTD region changed significantly. G_max/G_min markedly increased from 11.0 to 136.7, but AS degraded significantly from 4.95 to 5.18. Although G_max/G_min was sufficiently high for learning, AS was too high, which could significantly degrade the learning accuracy of the HW-NN. The learning accuracy with respect to the pulse condition is presented in fig. S6. A detailed description of the used datasets and the designed neural network is provided in the following section (Fig. 4). As a final step, we adjusted only the amplitude of the depressing pulse (step #3) after fixing the potentiating pulse to 5 V to improve AS and simultaneously secure a reasonable G_max/G_min value (Fig. 3D). The pulse width and frequency were fixed at 100 ms and 5 Hz, respectively. As the amplitude of the depressing pulse was varied from −5 to −2 V, the linearity of the LTP/D characteristic curve recovered slightly. In addition, AS was significantly improved from 5.18 to 4.77, which is expected to be a desirable pulse condition for achieving a very high learning accuracy of the HW-NN. Furthermore, we extracted the effective number of conductance states between G_max and G_min (NS_eff) from the LTP/D characteristic curves in Fig. 3D (see fig. S7). In the cases of 5 and −2 V pulses, 99 and 87% of the total conductance states were effectively accessible, which indicated that our synaptic device had a sufficient number of conductance states between G_max and G_min. The high operating voltage pulses are expected to be sufficiently reduced by (i) scaling down the device dimensions, (ii) adjusting the ion gel composition of the weight control layer (WCL), or (iii) replacing the WCL itself with ferroelectric materials including polyvinylidene difluoride, lead-zirconate-titanate, and hafnium zirconium oxide (HZO).

Fig. 3. Controllability and electrical stability of dynamic characteristics.

(A) Schematic illustration of the SIZO/ion gel synaptic device showing an application of the V_pre and V_WC, where the V_WC was varied according to three steps: step #1, the potentiating and depressing pulse amplitude variations from ±3 to ±5 V; step #2, the pulse width variations from 40 to 100 ms; and step #3, the depressing pulse amplitude variations from −5 to −2 V. (B to D) Normalized LTP/D characteristic curves according to the pulse number (top panel) and the extracted G_max/G_min and AS from the curves (bottom panel) corresponding to steps #1 to #3. (E) Electrical stability analysis of the dynamic characteristics with the optimal pulse condition (V_WC = +5/−2 V, 100 ms, 5 Hz). For the cycle test, a set of V_WC pulses consisting of 100 potentiating and 100 depressing pulses was applied to T_WC repeatedly 50 times (a total of 10,000 pulses). (F) Extracted LTP/D characteristic curves for the first cycle and every 10th cycle from 10th to 50th. (G) Extracted G_max/G_min and AS from the six LTP/D characteristic curves.

Fig. 4. Sensory-neuromorphic systems for sign language translation.

(A) Schematic illustration of the designed sensory-neuromorphic system for sign language translation consisting of stretchable resistive sensors and flexible artificial synapses. (B) Translation process via the designed sensory-neuromorphic system. Here, we assumed that a user wore a wearable glove on which a flexible sensory-neuromorphic system was embedded for communication with a nondeaf person. (C) Designed single-layer ANN with a size of 15 × 8 for the training and recognition simulation with respect to eight disparate hand signs of “A,” “B,” “C,” “I love you,” “H,” “E,” “L,” and “O,” where “T,” “I,” “M,” “R,” and “L” are the thumb, index finger, middle finger, ring finger, and little finger, respectively. Rows “1,” “2,” and “3” of the input image pattern refer to the distal interphalangeal joints, proximal interphalangeal joints, and metacarpophalangeal joints, respectively (see more detailed information in fig. S10). (D) Synaptic weight representation expressed by the conductance difference of two equivalent synaptic devices. (E) Mapping images of the normalized synaptic weights for the “I love you” hand sign before training and after finishing the 100th training epoch. (F) Confusion matrix between actual (input) and predicted (output) hand signs after the 100th training epoch. (G) Recognition rate according to the number of training epochs for the hand sign patterns with respect to the depressing pulse amplitude from −5 to −2 V.

We then investigated the electrical stability of the dynamic characteristics of the synaptic devices with the optimal pulse conditions (V_WC = +5/−2 V, 100 ms, 5 Hz). For the cycle test, a set of V_WC pulses consisting of 100 potentiating and 100 depressing pulses were applied to T_WC 50 times (a total of 10,000 pulses). As shown in Fig. 3E, the LTP/D characteristics of the flexible SIZO/ion gel synaptic device were maintained even after the 50 cycles without any significant degradation, exhibiting high stability under repeated electrical stress. We also extracted the LTP/D characteristic curve for the first cycle and every 10th cycle from 10th to 50th (Fig. 3F). The selected six LTP/D curves showed very similar curve shape, which indicates that our synaptic device could maintain its initial dynamic characteristics even after numerous weight update events for learning. To evaluate its electrical stability more precisely, we extracted G_max/G_min and AS from the six LTP/D characteristic curves (Fig. 3G). The obtained G_max and G_min values for 50 cycles remained constant at approximately 12.1 and 0.3 nS, respectively, and consequently, G_max/G_min was also maintained at approximately 44.8. Because of their high stability for the cycle, the RSD values of G_max, G_min, and G_max/G_min were also as small as 1.9, 5.7, and 5.1%, respectively (top panel of Fig. 3G). Moreover, the nonlinearities of the LTP and LTD curves (NL_P and NL_D) for 50 cycles did not change from their initial values of −1.83 and − 6.61, respectively, and the calculated AS value also showed no degradation at approximately 4.78 against the cycle test. The RSDs of NL_P and NL_D were 6.2 and 1.8%, respectively, and, in particular, that of AS was extremely as small as 1.2% (bottom panel of Fig. 3G). On the basis of the results, we confirmed that the dynamic characteristics of the synaptic devices had a very high stability for electrical stress. In addition, we investigated the reading and writing energy consumption of the synaptic device under potentiation/depression pulse conditions via measurements between T_pre and T_post and between T_WC and T_post (see fig. S8).

Sensory-neuromorphic systems for sign language translation

Last, we designed a sensory-neuromorphic system for sign language translation, consisting of stretchable resistive sensors and flexible artificial synapses (Fig. 4A). According to the World Federation of the Deaf, people who are hearing impaired and thereby use sign language as a first “language” account for almost 1% of the world’s population and have serious difficulty in communicating with normal people because of the deficiency in terms of language translation system (49). Therefore, it is essential to conduct research on the wearable sign language translation system that helps deaf people to communicate with nondeaf people without any delay or obstacles by providing direct and real-time sign language translation. The translation process via the designed sensory-neuromorphic system is as follows (Fig. 4B): (i) gesturing a certain word using sign language (“I love you” in this example), (ii) sensing the hand sign using stretchable sensors attached on each finger joint region, (iii) patterning (filtering, digitizing, and sampling, etc.) the sensed signals as a voltage form, (iv) inferencing the patterned data via a pretrained flexible synapse array, and (v) displaying the translated word on the screen or speaking it as a vocal sound (50). Here, we assumed that a user wore a wearable glove on which a flexible sensory-neuromorphic system was embedded for communication with a nondeaf person.

To evaluate the performance of the proposed sensory-neuromorphic system quantitatively, we theoretically constructed a single-layer ANN with a size of 15 × 8 and conducted training and recognition simulation for eight disparate hand signs of “A,” “B,” “C,” “I love you,” “H,” “E,” “L,” and “O” (Fig. 4C). The somewhat small size of the ANN is attributed to the limited number of finger joints (only 15 joints), and it can be expanded further to a larger network by adopting a sensor array onto the joints (51–54). For the simulation, we prepared 60,000 training and 10,000 testing patterns consisting of eight types of hand signs, where each hand sign pattern has a size of 3 × 5 (the detailed sensing and patterning process is shown in figs. S9 to S11). The training process was as follows: The 15 voltage values in the hand sign pattern are applied one-to-one to the 15 input neurons as input voltages (V₁ to V₁₅), and they were then transformed to the output currents (I₁–I₈) via the matrix-vector summation with a synaptic weight matrix (W_1,1–W_15,8). Here, synaptic weights can be expressed by the conductance difference between two equivalent synaptic devices, because the conductance of the hardware synaptic device is always positive, unlike that of the synapses in software with both positive and negative values (Fig. 4D) (55). Therefore, to map the synaptic weights of the designed fully connected HW-NN, a synaptic array of at least 15 × 8 × 2 is required. Next, the output values (O₁ to O₈) are calculated using the softmax activation function [O_m = f(I_m)], and they are further compared to the label value of the output neuron (k₁–k₈) to determine the sign of the weight update [sgn(∆W)]. Last, all weight values in the synaptic weight matrix are updated according to the determined signs (see Materials and Methods for details). We repeated this training process with 60,000 sets of training data and calculated the recognition rate with 10,000 sets of testing data every 60,000th training step (one training epoch) for 100 training epochs.

As shown in Fig. 4E, the mapping image of the normalized synaptic weights for the “I love you” hand sign had an uncertain shape before training, and the values were randomly distributed from −1 to +1. After finishing the 100th training epoch, the mapping image presented a specific pattern, which indicates that the ANN was well trained for the input hand signs. Here, weight mapping was conducted using a pattern recognition simulator that could monitor the synaptic weights at each training task step. The confusion matrix between the actual (input) and predicted (output) hand signs also showed that the ANN was trained almost perfectly after the 100th training epoch (Fig. 4F). Although the worst confusion case happened between the signs “A” and “O,” the sensitivity of each sign was still as high as 97.4 and 99.4%, respectively. Here, the sensitivity was calculated as the percentage of the predicted outputs over the total actual inputs of the corresponding hand sign. We then investigated the recognition rate according to the number of training epochs for hand sign patterns (Fig. 4G). This training and recognition simulation was repeatedly conducted using the LTP/D characteristics of Fig. 3D with respect to the depressing pulse amplitude from −5 to −2 V (see also fig. S6). As shown in the top panel of Fig. 4G, the recognition rate of the ANN increased rapidly as the training continued and was saturated after the 10th training epoch in all pulse amplitude cases. The highest recognition rate was obtained in the case of a −2 V depressing pulse amplitude (optimum pulse condition); in particular, its maximum value reached 99.4% (bottom panel of Fig. 4G). Such a high recognition rate was possible because the softmax function, which amplifies small differences between the input patterns by using an exponential term and distinguishing between the difference clearly, was adopted. Figure S12 presents the results of the recognition rate with respect to the bending radius and the bending cycle; these results indicate an excellent recognition rate exceeding 90%, even under physical bending conditions. For reference, based on the recognition tasks, the required device specification was estimated as follows: For achieving a recognition rate greater than 90%, AS below 5.0 and G_max/G_min above 10 are required. As shown in table S2, we additionally performed pattern recognition tasks by using the NeuroSim + MLP simulator (56) and compared the proposed synaptic device with other previously reported devices in terms of the recognition rates for the Modified National Institute of Standards and Technology dataset. Those results clearly demonstrate the feasibility of the sensory-neuromorphic system concept for sign language translation.

DISCUSSION

In this study, we successfully implemented a flexible artificial synapse on a SIZO/ion gel hybrid structure directly fabricated on a flexible PI substrate, in which the channel conductance was effectively modulated by ion movement in the ion gel. This was possible because of the low-temperature deposition process of the SIZO channel layer (<150°C) and the improvement of the surface roughness by the PVP buffer layer (12.29→1.81 nm after inserting a 3 wt % PVP layer). Our flexible synaptic device exhibited excellent mechanical flexibility and durability, where the static and dynamic characteristics were maintained stably when bending with a radius of 5 mm and even after 1500 bending cycles. Moreover, its LTP/D characteristics, such as G_max/G_min and AS, which were optimized by the V_WC pulse adjustment, were sustained without any degradation even after the application of 10⁴ V_WC pulses. Last, we proposed a novel concept of a sensory-neuromorphic system for sign language translation consisting of stretchable-resistive sensors and flexible artificial synapses, which could provide direct and real-time sign language translation to deaf people. The feasibility of the proposed system was confirmed via the training and recognition simulation using the hand sign patterns obtained by the stretchable sensors, and a maximum recognition rate of 99.4% was achieved using the optimized LTP/D characteristics. Moreover, an excellent recognition rate exceeding 90%, even under physical bending conditions, was demonstrated. We expect that this result will become an important basis of research on flexible artificial synapses for future wearable sensory-neuromorphic systems.

MATERIALS AND METHODS

Formation of the SIZO channel layer

A SIZO thin film with an approximate thickness of 30 nm was deposited as the active channel layer with a shadow mask in a radio frequency magnetron sputtering system. The SIZO ceramic target was prepared using high-purity (99.99%) powder mixtures of SiO₂, In₃O₂, and ZnO. Si was incorporated into the IZO (In:Zn = 7:3) system at 1 wt %. During the deposition process, the sputtering power, deposition pressure, and Ar flow rate were 40 W, 4 mtorr, and 40 standard cubic centimeter per minute (sccm), respectively. After the deposition, the SIZO thin films were annealed at 150°C for 2 hours in a furnace.

Fabrication of the flexible SIZO/ion gel synaptic device

The PVP solution was prepared by dissolving PVP powder in propylene glycol methyl ether acetate (Sigma-Aldrich). Next, the PVP solution was spin coated onto a PI film with a size of 2 × 2 cm² at 4000 revolutions per minute using a spin coater (ACE 200). The PVP-coated substrate was annealed on a hot plate at 100°C for 10 min and at 200°C for 2 min and 30 s. To form the T_pre, T_post, and T_WC, the first shadow mask was used on the PVP-coated PI substrate, followed by the deposition of 5-nm-thick Ti and 20-nm-thick Au using an electron beam evaporator. The SIZO active channel layer was deposited between T_pre and T_post with the second shadow mask. The distance between the two terminals and the width of the terminals were 30 and 30 μm, respectively. Then, the ion gel was patterned by exposing it to ultraviolet light for 40 s through the third shadow mask.

Characterization of SIZO/ion gel synaptic device

The OM images of the SIZO/ion gel synaptic devices were obtained using an upright metallurgical microscope (Olympus BX53M). For surface analysis of the synaptic device, AFM measurements were performed at room temperature under ambient conditions using a platinum/iridium–coated Si tip in an NX10 system (Park Systems Corp.). The RMS_roughness value was calculated using the following equation

$${\text{RMS}}_{\text{roughness}}=\sqrt{\frac{1}{A}{\iint }_A{Z}^2(x,y)\mathrm{d}x \mathrm{d}y}$$

For analysis of the I_post-V_WC and LTP/D characteristics of the SIZO/ion gel synaptic devices, electrical measurements were conducted at 20°C and a relative humidity of 40% (monitored by a commercial thermo-hygrometer) using a Keysight B2901A parameter analyzer. To avoid the measurement errors appearing below the minimum source current resolution of 10⁻¹² A, all the synaptic characteristics, including V_th, G_post, and G_max/G_min, were extracted from the raw I-V data above 10⁻¹⁰ A.

Fabrication of the CNT/SEBS stretchable sensor

A polystyrene-block-poly(ethylene-ran-butylene)-block-polystyrene (SEBS) H1062 elastomer (Asahi Kasei Company) substrate was prepared, where SEBS H1062 (200 mg/ml in toluene; Sigma-Aldrich) was cast onto a petri dish to form elastic films with an approximate thickness of 0.2 mm. CNTs (U.S. Research Nanomaterial) dispersed in 10 ml of toluene at a concentration of 1.5 wt % were then ultrasonicated for 30 min. The CNT solution was coated on the SEBS substrate, and the toluene was evaporated in a fume hood to embed the resistive sensing material onto the substrate.

Weight update in the synaptic device for pattern recognition tasks

To update the synaptic weight (W_n,m), the sgn(∆W) was calculated by subtracting the output value to the corresponding label value [i.e., sgn(∆W) = k_m − O_m]. If sgn(∆W) is positive (positive feedback), then G_n,m⁺ increases and, simultaneously, G_n,m⁻ decreases, causing the potentiation of the synaptic (W_n,m↑ = G_n,m⁺↑ − G_n,m⁻↓). In contrast, if sgn(∆W) is negative (negative feedback), then the synaptic weight is depressed (W_n,m↓ = G_n,m⁺↓ − G_n,m⁻↑). If sgn(∆W) was zero, then the synaptic weight was not updated. All weight values in the synaptic weight matrix were updated according to the determined sgn(∆W). Here, the conductance change (ΔG) was determined using the following equations

$$\begin{array}{c}{G}_{n+1}=G_n+\mathrm{\Delta }{G}_{\mathrm{P}}=G_n+{\mathrm{\alpha }}_{\mathrm{P}}{e}^{-{\text{NL}}_{\mathrm{P}}\frac{G_n-G_{\text{min}}}{G_{\text{max}}-G_{\text{min}}}} (\mathrm{\Delta }\mathit{G}>0,G\uparrow ),\\ G_{n+1}=G_n+\mathrm{\Delta }{G}_{\mathrm{D}}=G_n-{\mathrm{\alpha }}_{\mathrm{D}}{e}^{{\text{NL}}_{\mathrm{D}}\frac{G_{\text{max}}-G_n}{G_{\text{max}}-G_{\text{min}}}} (\mathrm{\Delta }\mathit{G}<0,G\downarrow )\end{array}$$

In these equations, G_{n + 1} and G_n denote the synaptic conductance when the n + 1th and nth pulses are applied, respectively, and parameters α and NL denote the amount of conductance change and the nonlinearity, respectively. Details regarding the estimation of the α and NL values are provided in fig. S13.

Hand sign language translating simulation in sensory-neuromorphic system

The hand sign language translating simulation was conducted on the basis of a single-layer perceptron (SLP) ANN, where the ANN had a size of 15 × 8 and was theoretically constructed using nonideal factors, such as G_max/G_min and AS. The input voltage values for the hand sign patterns were normalized to 256 grayscale values from 0 to 1 V, and the softmax function was used as the activation function. After training the SLP ANN with 60,000 patterns (one training epoch), the recognition tasks for the 10,000 separate testing patterns were performed. The recognition rate was calculated at every 60,000th training step during the training process.

Supplementary Materials

This PDF file includes:

Figs. S1 to S13

Tables S1 and S2

References