Fig. 4. Sensory-neuromorphic systems for sign language translation.

(A) Schematic illustration of the designed sensory-neuromorphic system for sign language translation consisting of stretchable resistive sensors and flexible artificial synapses. (B) Translation process via the designed sensory-neuromorphic system. Here, we assumed that a user wore a wearable glove on which a flexible sensory-neuromorphic system was embedded for communication with a nondeaf person. (C) Designed single-layer ANN with a size of 15 × 8 for the training and recognition simulation with respect to eight disparate hand signs of “A,” “B,” “C,” “I love you,” “H,” “E,” “L,” and “O,” where “T,” “I,” “M,” “R,” and “L” are the thumb, index finger, middle finger, ring finger, and little finger, respectively. Rows “1,” “2,” and “3” of the input image pattern refer to the distal interphalangeal joints, proximal interphalangeal joints, and metacarpophalangeal joints, respectively (see more detailed information in fig. S10). (D) Synaptic weight representation expressed by the conductance difference of two equivalent synaptic devices. (E) Mapping images of the normalized synaptic weights for the “I love you” hand sign before training and after finishing the 100th training epoch. (F) Confusion matrix between actual (input) and predicted (output) hand signs after the 100th training epoch. (G) Recognition rate according to the number of training epochs for the hand sign patterns with respect to the depressing pulse amplitude from −5 to −2 V.