A 3D-printed neuromorphic humanoid hand for grasping unknown objects

Summary

Compared with conventional von Neumann’s architecture-based processors, neuromorphic systems provide energy-saving in-memory computing. We present here a 3D neuromorphic humanoid hand designed for providing an artificial unconscious response based on training. The neuromorphic humanoid hand system mimics the reflex arc for a quick response by managing complex spatiotemporal information. A 3D structural humanoid hand is first integrated with 3D-printed pressure sensors and a portable neuromorphic device that was fabricated by the multi-axis robot 3D printing technology. The 3D neuromorphic robot hand provides bioinspired signal perception, including detection, signal transmission, and signal processing, together with the biomimetic reflex arc function, allowing it to hold an unknown object with an automatically increased gripping force without a conventional controlling processor. The proposed system offers a new approach for realizing an unconscious response with an artificially intelligent robot.

Graphical abstract

Highlights

• •3D curvilinear structures were fabricated by multi-directional 3D printing
• •A Kresling-based origami sensor was designed for precise pressure sensing
• •Our neuromorphic hand shows the bioinspired signal perception
• •The neuromorphic humanoid hand demonstrates unconscious grasping of unknown objects

Computer hardware; Manufacturing; Robotics; Automation; Sensor system

Introduction

Neuromorphic systems are used to mimic the biological signal perception process of the biological system because of the ultrahigh speed of parallel operation for spatial and temporal information (Fu et al., 2018; Han et al., 2015; Mead, 1990; Zhu et al., 2014). Several generations of neuromorphic systems have been developed including algorithm-based neuromorphic computing and hardware-based neuromorphic circuit for emulating parallel calculations (Tuchman et al., 2020). Because both of them depended on central processing units, graphics processing units and tensor processing units still faced the high energy consumption concern for the traditional von Neumann’s architecture and limitations of Moore’s law (Berggren et al., 2020; Moore, 1965; Xia and Yang, 2019). Recently, third generation of “in-memory computing” based neuromorphic sensing systems were reported by mimicking the biological signal perception function (Berggren et al., 2020; Xia and Yang, 2019). The biological signal perception function includes environmental sensing, signal transmission, and signal processing. Different types of neuromorphic sensing systems with signal perceptions functions have been developed to mimic the sensing functions, such as vision, touch, and taste sensing (Bao et al., 2021; Berridge et al., 2000; Kyung et al., 2015; Lee and Lee, 2019).

An ideal neuromorphic sensing system not only demonstrates signal perception but also controls artificial effectors such as humanoid hands by refining a control procedure (Tuchman et al., 2020). An artificial afferent nerve with a pressure sensor was demonstrated for the control of a cockroach leg (Kim et al., 2018). Also, the vision-based optoelectronic sensorimotor can manipulate artificial muscle based on light cognition (Lee et al., 2018). In the biological system, there are two different control mechanisms; a brain processed conscious control and a spinal cord processed unconscious control. In some biological systems, not all environmental stimuli are processed by the brain (He et al., 2020). Instead, the spinal cord produces a faster unconscious response as a simple reflex arc, thereby reducing the brain’s tasks. Recently, the concept of unconscious controlling was replicated through an artificial somatic reflex arc system (He et al., 2020). Moreover, a conscious response has been demonstrated using an artificial stimulus-response system, and the response time was reduced through training (S. Kim et al., 2021a). However, the effect of training on the unconscious response of neuromorphic systems has not been studied yet.

In the field of pressure sensing, artificial skin has been highlighted as a type of biomimetic touch sensors. Different types of pressure sensors have been proposed for the sensor parts, including capacitance-based and resistance-based sensors (Chortos et al., 2016). A force-sensitive resistor (FSR)-based sensor has been used as a pressure sensor, because of its low cost, small size, and high sensitivity to force (Gupta et al., 2011). However, the sensing mechanism of FSR, deformation of active layer or diaphragm depending on the force, generates concerns such as its non-linearity and non-repeatability. Such concerns are considered as evaluation criteria of the sensors (Abdul Razak et al., 2012; Kumar and Pant, 2014). Therefore, there is a need for a force-sensitive resistor without deformation of the active layer. Utilizing the sensing mechanism of a potentiometer is a possible solution, as it shows a variable resistance change without physical deformation of the active layer by changing the length of the conducting path. The remaining challenge is to develop a class of variable resistors that change axial forces into rotational forces. The 3D origami structures have been demonstrated with their rotational multifunctionalities (Li et al., 2019, 2020). In particular, the Kresling origami, formed by a series of tessellated triangles, shows rotating characteristics when compressed (Zhai et al., 2018). In addition, its rotating behavior can be tuned by modifying the angular design of the tessellated triangle(s), allowing for customization of the force sensing (Novelino et al., 2020; Zhai et al., 2018). Applying the origami structure to the pressure sensor allows for force conversion without requiring other components, such as gears or axial shifters. Indeed, employing such a structure simplifies the entire pressure sensing system for various applications (Kaur et al., 2021).

For the fabrication of three-dimensional (3D) structural components, 3D printing has been widely adopted for prototyping because of its advantages in regard to custom and facile fabrication. Various 3D printing technologies have been developed, such as fused filament fabrication (FFF), digital projection lithography, direct ink writing (DIW), and selective laser sintering. Among these, the DIW method for paste printing has attracted significant attention for the preparation of functional materials (Dong et al., 2018; Kim et al., 2019; Lewis, 2006; Valentine et al., 2017; Zhang et al., 2019; Zhou et al., 2017). DIW-printed structures are mainly dependent on the properties of the printed materials, whereas in-situ sintering with a laser facilitates the fabrication of 3D structures. For example, freestanding structures have been achieved using laser-assisted methods (Skylar-Scott et al., 2016). Circuits made from copper have been demonstrated on a 3D surface using a laser-assisted DIW method (Jo et al., 2020). However, 3D printing on curvilinear surfaces while keeping the nozzle vertical to the surface is difficult in the typical DIW because of the limited freedom of current three-axis DIW printers. One unique solution is to add more freedom of motion to the printers by using a multi-axis robot arm. Robot arms are widely used in the industry for more complex applications, such as welding for automobiles. Nonetheless, to the best of our knowledge, this work is the first DIW printing method with a six-axis industrial robot arm. We used a DIW approach using a multi-axis robot arm, as shown in Figure 1A. The six-axis robot arm enables the seamless fabrication of conductive traces on vertical and tilted surfaces in a single printing process. Moreover, the 3D-printed conductor can be sintered in situ using an infrared laser (IR) laser with optimized parameters, as shown in Figure 1B. Unlike conventional three-axis 3D printing systems, the demonstrated six-axis robot 3D printing can fabricate 3D electronics on curvilinear surfaces.

Figure 1.

3D printing through the six-axis robot

(A) Schematic diagram of robot 3D printing for three different types of surfaces.

(B) Schematic diagram of printing procedure with in situ laser sintering.

(C) The six-axis robot 3D printing setup.

(D) Image of the laser spot and sintering (top: in-situ scanning; bottom left: as-printed conductive trace; bottom right: sintered conductive trace).

(E) Relation between pulse width and induced temperature at a frequency of 100 Hz. Data are represented as mean ± SEM.

(F) Resistance changes with scanning at a speed of 4 mm/s.

(G) Relation between scanning speed and sintering efficiency. Data are represented as mean ± SEM.

(H) Left: sample printed by six-axis robot on the vertical wall of a 3D-shaped oscillator. Right: design of 3D-shaped oscillator.

(I) Left: the kirigami conductive path on a 3D origami finger using six-axis robot 3D printing. Right: design of the kirigami conductive path on the tilted surfaces.

Here, we demonstrated a neuromorphic humanoid hand, fabricated by six-axis robot-based DIW method and integrated with origami-inspired pressure sensors. A portable neuromorphic system is added for the bioinspired signal transmission and processing without conventional spike-based control algorithm. The 3D structural neuromorphic humanoid hand is trained to demonstrate a reflex arc by gripping an object with an unconsciously increased force, based on the learning and non-volatile memory functions of the neuromorphic system.

Results

3D-printed structural components

Multi-axis robot arms are widely used in industries, as they are designed with several controllable freedoms. We integrated DIW printers with a six-axis robot arm. It is a promising solution for solving the shortage of current DIW printers that can vertically print features on planar surfaces. Also, an in situ thermal processing with IR laser or UV light has been demonstrated for the paste-based DIW (Skylar-Scott et al., 2016). Here, we utilized an IR laser-assisted six-axis robot printing system for the fabrication of 3D structural components as shown in Figure 1C. The demonstrated system comprises a six-axis robot, IR power control system, and DIW dispenser. A power control system constructed with an IR laser and function generator is used to manage the applied laser for sintering the printed conductive traces. The DIW nozzle and IR lasers are mounted on a printed head together and controlled by the robot. The bottom-left image Figure 1D shows the as-printed conductive traces, whereas the right-side image shows the fully sintered sample. The sintering level depends on the heat generated by the IR laser and is modulated by the function generator, including the pulse width and power. As shown in Figure 1E, the maximum temperature of the substrate increases with the pulse width at a fixed frequency and laser power. The sintering temperature is 121.5°C under a pulse width of 1 ms, whereas it increases to 174.4°C under a pulse width of 2 ms. This is because a longer pulse width increases the duration of the laser spot on the sample, resulting in a higher temperature. To examine the sintering effect, the resistance change of the printed line is monitored at a printing speed of 4 mm/s, as shown in Figure 1F and Video S1. The resistance drastically decreases from 40 s onward, because of the partial sintering. After 80 s, the printed line is fully sintered and the resistivity has reached a stable value. The efficiency of the sintering is illustrated in Figure 1G, where it is expressed as the total duration time per millimeter. It is found that increasing the scanning speed improves the sintering efficiency.

The printing route of the DIW printing head is controlled by the six-axis robot arm. Because of the six degrees of freedom, the robot system can be used to print on different types of surfaces. The right side of Figure 1H is a schematic of a 3D-shaped oscillator fabricated for the portable neuromorphic system. Conductive traces are designed on each vertical wall. In a conventional DIW printer, the vertical walls must be placed upward before further printing. Repeating these procedures until the completion of printing all of the other walls requires energy. Seamless printing is possible using a proposed six-axis robot-integrated DIW printer, while keeping the nozzle consistently vertical to the walls, as shown in Video S2. Furthermore, tilted surfaces can also be fabricated by the proposed printing, as shown in Figure 1I. The 3D Miura-ori origami finger has serpentine valleys on its surface (T. H. Kim et al., 2021; Xu et al., 2017). The multi-axis robot printing process allows for the fabrication of a complex kirigami conductive path on the 3D origami finger. Kirigami patterns are well known as stretchable structures against bending behaviors (Xu et al., 2017). Therefore, both the 3D origami fingers and kirigami conductive path provide strength for the repeated bending condition. Indeed, the kirigami conductive path shows a synergic relation with the 3D origami module, as shown in Video S3. Through in situ laser sintering, the printed conductive traces on the vertical walls and tilted surfaces are seamlessly fabricated, as shown on the left sides of Figures 1H and 1I.

Pressure-sensitive humanoid hand

The origami finger comprises an array of 3D Miura-ori structures for the humanoid robot hand. The 3D origami finger was optimized based on two different criteria: 1) maximum bending in one direction and 2) minimized plasticity after large-cycle bending. A central composite design based on a statistical method was used for the optimization process (T. H. Kim et al., 2021b). The optimized finger frames were assembled as shown in Figure 2A. A pressure sensor with variable resistance is integrated on the fingertip. The Kresling origami was used as an active sensor frame, as shown in Figure 2B. The resistor length was changed between the 1^st and 2^nd paths, depending on the rotation degree of the origami structure (Novelino et al., 2020). Thus, the resistance of the conductive path changed linearly. When the 3D origami was compressed, the resistor length between 1^st and 2^nd paths was decreased, because of the simultaneous rotation of the origami. So, the resistance decreased as shown in Figure 2C. In addition, the rotating feature of the 3D origami was tunable by changing the angular designs (α and β angles) of repeated triangles in the origami (Novelino et al., 2020; Zhai et al., 2018). Smaller angles in the design generate less structural rigidity of the entire origami structure. So, the angles of 30° and 38° for α and β, respectively, have been selected as the minimum boundary parameters based on their geometrical properties. More details are discussed in Figure S1A. At different angles of β = 38°–41°, a higher value of β shows a more rotatable behavior under compression, as shown in Figure 2D. Among other angular designs, the higher linearity of rotation under compression reflects the optimal design, as a larger rotating degree reflects a larger length change of the conductive path. Thus, the linearity of the rotated angle at different angular designs could be estimated using Equation 1.

$$\text{Nonlinearity}\left(\%\right)=\frac{D_{in\left(max\right)}}{I{N}_{f.s.}}\times 100$$ (Equation 1)

Figure 2.

3D-printed humanoid hand with 3D origami pressure sensor

(A) Schematics of the humanoid robot hand with 3D origami fingers equipped with the pressure sensor on the fingertip.

(B) Variable resistance sensor developed by 3D Kresling origami.

(C) The rotating behavior of the origami and its resistive length changes under compression.

(D) The rotating angles of the 3D origami at different angular designs.

(E) The resistance changes of 3D origami pressure sensor under compression. Data are represented as mean ± SEM.

(F) Resistance changes before and after the gripping a ball.

Here, D_in(max) and IN_f.s. are the maximum strain deviation and strain at the full scale, respectively. The D_in(max) has been determined by measuring the maximum difference between actual curves and their ideal linear lines to the endpoints of the graphs indicated by the dotted line in Figure 2D (Regtien and Dertien, 2018; Carr and Brown, 2000). The nonlinearity values of the 3D origami with β = 38°, 39°, 40°, and 41° are 6.74%, 6.42%, 6.20%, and 7.09%, respectively. The angle of α is fixed at 30°. Among the range of the angle parameters, the 3D origami with β = 40° shows the lowest nonlinearity; thus, it was selected as the final design for the 3D origami sensor. Then, the conductive path (R: 3.8 × 10³ Ω∙m) was printed on the top of the inner wall of the 3D origami. The resistance changes at different compressions are shown in Figure 2E. When the 3D origami sensor was compressed, a change in the resistance was observed, as shown in Figure S1 and Video S4. The sensitivity of the 3D origami sensor was calculated using Equation 2, as follows:

$$\text{Sensitivity}=\frac{\text{Δ}{R}_{max}}{\text{Δ}{S}_{ma{x}_{}}}$$ (Equation 2)

Here, ΔR_max is the resistance difference between the original and compressed states of the 3D origami, and ΔS_max indicates the maximum compressed length of the origami. The calculated sensitivity of the sensor is 53.13 Ω/mm. In addition, the gauge factor (GF), defined as GF = |ΔR/R₀|/ε, was evaluated, where ε and ΔR are the strain of the sensor and changed resistance after compression, respectively. A GF of 2.5 was obtained from the 3D origami at a 35% strain. GF is one of the representative parameters to define the sensitivity of sensors. Therefore, we compared the GF of our origami sensor with that of other sensors as shown in Table 1 (Garcia et al., 2021). Our sensor does not show the best performance among them, but it has potential to be improved because its mechanism relies more on the structure variation than the material property. Therefore, it is expected that the GF of the sensor can be further improved when the structure is further optimized. After integrating the 3D origami sensor into the humanoid robot hand, we investigated the pressure sensing performance of the hand as shown in Figure 2F. Before gripping, the highest resistance was observed, and when the robot hand started to grip a ball, the compressed 3D origami sensor on the fingertip caused the resistance to decrease drastically. The resistance changes before and after gripping are presented in Video S5. Thus, we demonstrated a pressure sensing robot hand. In addition, the kirigami conductive path (Xu et al., 2017) on the origami finger was used to integrate the pressure sensing fingertip and multimeter, minimizing the wire integration. Thus, we secured additional space for more functionalities with the pressure sensing performance of the robot, and potential problems such as disconnection between the sensors and processor were prevented during the bending of the finger; this was attributed to the six-axis robot DIW printing on the complex origami surface of the finger. Thus, a sensory humanoid robot hand was prepared with the pressure sensing function.

Table 1.

Various types of strain sensors and their corresponding gauge factors

Mechanism	Type	Materials	G.F.	Ref.
Piezoresistive	Axial strain	Graphene–nanocellulose composite nanopaper	7.1 at 100% strain	(Yan et al., 2014)
		Silver nanowire-PDMS Nanocomposite	5 at 60% strain	(Amjadi et al., 2014)
		ZnO nanowire/polystyrene-hybridized flexible films	116 at 50% strain	(Xiao et al., 2011)
		Thermoplastic polyurethane (TPU)-boron nitride nanosheets composite	7.9 at 60% strain	(Tan et al., 2020)
Capacitative	Origami folding	Commercial conductive Electrif	0.72 at 40% strain	(T. H. Kim et al., 2021b)
Capacitative	Wrinkled degree	Wrinkled ultrathin gold films	3.05 at 140% strain	(Nur et al., 2018)
Potentiometric	Origami folding	3D-printed silver paste	2.5 at 35% strain	(Our works)

3D-integrated neuromorphic humanoid hands

For neuromorphic robot systems with complete biological sensory neuron network sequences, three different types of functions are required: sensing, signal transmission, and signal processing functions. Therefore, a portable neuromorphic system, including signal transmission and processing, was developed and integrated with the fabricated robotic hand as a neuromorphic humanoid hand, as shown in Figure 3A. The pressure sensors on the fingertips were connected to the neuromorphic system as the input stimuli. The portable neuromorphic system was built with a 3D-shaped oscillator and zinc-tin-oxide (ZTO)-based synaptic transistor for bioinspired signal transmission and processing (Bao et al., 2021). The 3D Colipits oscillator converts DC signal to AC signal as spikes in biological systems. As shown in Figure S2, the 3D oscillator and synaptic transistor were mounted on a printed circuit board. All of the electronic components of the neuromorphic system were designed to be replaceable and modulated into a single 3D structural board; as such, they were considered as a portable neuromorphic system. A portable design for electronic devices allows minimizing the additional integration required for robot applications with easier maintenance.

Figure 3.

Neuromorphic humanoid hand

(A) Left: design and image of the integrated neuromorphic humanoid hand; Right: details of the 3D portable neuromorphic system, constructed by a 3D-shaped oscillator and a synaptic transistor.

(B) Relation between the amplitude of the output signal from the 3D-shaped oscillator and resistance. Data are represented as mean ± SEM.

(C) Non-volatile memory effect of synaptic transistor.

(D) Synaptic current change with variable force. (E) Synaptic current change with a constant force.

In this system, the received signal undergoes a few steps on the demonstrated neuromorphic humanoid hand. First, the force detected by pressure sensors on the fingertips is transmitted to the neuromorphic system, followed by its conversion into spike-form signals such as those in biological systems by the oscillator. As shown in Figure S3A, the pressure sensor is connected between the power source and gate electrode of the transistor. Thus, the pressure-induced resistance change affects the actual power supplied to the gate, resulting in variations in the output AC signal from the oscillator. The amplitude significantly decreases when the resistance is large. A similar experimental result is obtained as shown in Figure S3B, and as summarized in Figure 3B. The amplitude drops from 1.6 V to 0.8 V when the resistance is increased from 25 Ω to 100 Ω. Similar results are obtained when the pressure sensor is connected to the source electrode of the transistor, as shown in Figures S3C and S3D. Second, the performance of synaptic transistors is confirmed before processing the converted signal. Because synapses accomplish memory and learning functions in biological systems, synaptic transistors work as artificial synapses for replicating these functions. As shown in Figure 3C, two electrodes are fabricated on a ZTO semiconductive layer. Moreover, to constrain the hydrophilic electrolyte just above the electrodes, a circular wall is fabricated with the facilitation of a digital camera magnifier for accurate nozzle alignment, as shown in Figure S4. The non-volatile memory and learning behaviors of synaptic transistors are accomplished by ion-accumulation at interfaces and ion-doping to the semiconductive layer (Bao et al., 2021). When a positive bias is applied to the gate of synaptic transistors, ions in the gate electrolyte will accumulate at the interface, so some ions will dope into the semiconductor layer, resulting in an increased conduction of the channel. When a larger bias or a bias with longer time is applied, conductance can be largely increased. As a result, under a continuous pulse positive bias, the postsynaptic current increases with time, as shown in Figure 3C. Finally, the converted signal is transmitted to the synaptic transistor for processing. As mentioned above, higher pressure on the pressure sensor results in lower resistance, leading to a larger output signal from the oscillator. Then, the output spike signal is applied to the gate of the synaptic transistor of the integrated neuromorphic system. In general, a higher positive bias on the synaptic transistor results in a larger current, as more ions accumulate at the interface and are doped inside the semiconductive channel. The entire signal flowing is illustrated in Figure S5. This phenomenon can be confirmed using the integrated neuromorphic systems, as illustrated in Figure 3D. An increased force boosts the postsynaptic current because of smaller resistance under a larger force, and vice versa. Moreover, when applied to a constant force, the postsynaptic current also increases with time because of the non-volatile effect, as shown in Figure 3E. The non-volatile effect lasts until the force is retrieved.

Reflex arc function through neuromorphic humanoid hands

In biological systems, after receiving a signal from a sensory neuron network, the brain makes a decision as a response to environmental stimuli to actuate organs. However, another type of unconscious response is also performed through reflex arcs in biological systems, without a neural response from the brain (He et al., 2020). A bioinspired neuromorphic humanoid hand must be equipped with such functions to act as an artificially intelligent robot.

In general, an environmental stimulus can be captured, transmitted, and processed by integrating the pressure sensor with the demonstrated neuromorphic system. However, to fully replicate a bioinspired reflex arc, the actuation function needs to be realized as an unconscious response. In general, the degree of unconscious responses can be trained. For example, the gripping force for one person to grip an unknown object the first time is different from the force to grip it in a second time. This is because experience is obtained by gripping the unknown object where the previous experience works as a training step. Thus, a suitable force can be applied for secondary gripping after training. In other words, the experience obtained through training influences the degree of the reflex arc. Therefore, by realizing the bioinspired reflex arc function for an unconscious response, the fabricated neuromorphic humanoid hand becomes a bio-inspired artificially intelligent robot. Figure 4A illustrates a strategy for this artificially intelligent robot to grip objects by training. First, the grasping force is detected by the pressure sensor on the fingertips. Then, the signal will be transmitted and processed through the oscillator and synaptic transistor in the neuromorphic system, respectively. Then, the output from the neuromorphic system is used to drive the motors, which manipulate the gripping motion of the fingers on the humanoid hand. A larger output from the neuromorphic system will produce a higher voltage on the motors to create stronger grasping force. Before training, the initial gripping force based on previous experience is not strong enough to hold the object because the synaptic transistor is in its initial low conductive state. However, after a few training cycles, the neuromorphic system increases the gripping force unconsciously by sufficient force to hold the object because of the increased conduction of the synaptic transistor. This control mechanism is hypothesized from the increased gripping force automatically without signal processing through a central microprocessor.

Figure 4.

Unconscious response through artificially intelligent robot

(A) Schematic description of the unconscious response.

(B) Relation of the output voltages from the oscillator under repeated trainings, and output voltages from the neuromorphic system under repeated trainings.

(C) Quantitative validation of increased force from the pressure sensor’s displacement (indicated as the red line in the inserted image) and the corresponding gripping force at certain voltage output from the neuromorphic system. Data are represented as mean ± SEM.

(D) Images of gripping motions: Slipping a ball before training, 1^st training, 2^nd training, 3^rd training, and holding of a ball.

The proposed strategy is verified through the following steps. First, the robot hand is trained three times using the same force. As shown in Figure 4B, the output from the oscillator is nearly the same in each case, as the force on the pressure sensor is constant as a repeated input for each training. The response time of the sensor has been measured. The required time for a sensor signal to change from 0 to 63.2% or 90% of the full scale is considered at instantaneous full-scale pressure change of the sensor. The response times for 63.2% and 90% changes of the sensor signals are 8.2 ± 3.52 ms and 12.4 ± 7.92 ms as shown in Figure S6, respectively, after pressure release. However, the motor driving voltage from the neuromorphic system is gradually increased with the training time owing to the non-volatile effect mentioned above, as shown in Figure 4C. When the synaptic transistor was trained three times by a full scale of deformation in the pressure sensor, the output voltage of the neuromorphic system applied to the motor was increased from 0.8 to 7.4 volts, which made the motor produce a potent force to grab a ball. Figure 4D illustrates the detailed training procedure for the neuromorphic humanoid hand by grasping an unknown object. Before training, the ball is dropped, because of the weak gripping force. After the first training, the bending of the finger is larger than the initial state and creates a stronger touch on the ball, thereby representing a stronger gripping force. After two more trainings, the depth of touch is increased to produce a much stronger gripping. Finally, the motor is driven to produce sufficient force to hold the ball, as shown in Video S6. The increased gripping force produced during the training is quantitatively validated through the pressure sensors integrated on fingers as shown in Figure 4C. It indicates a relation between the compressing force and the relative displacement of the fingertips when the pressure sensor is compressed, which is measured and averaged from 12 measurements by a universal testing machine (Shimadzu EZ-LX). The resulting gripping force is quantitatively described in the bottom graph of Figure 4C together with the displacement of each training state, which is the compressed depth. Meanwhile, the output voltage from the neuromorphic system after each training state can also be found in Figure 4B. Therefore, the relation between the neuromorphic system’s output and the gripping force is described in Figures 4B and 4C. At the initial state before training, a lower output voltage was generated from the neuromorphic system resulting in a smaller gripping force that is confirmed by the small displacement of fingertips as shown in the bottom graph of Figure 4C. The output voltage and gripping depth were increased gradually with the increased number of trainings. Moreover, because the output voltage from the neuromorphic system can also be modulated by varying amplitude and duration of each training state, the gripping force can also be adjusted when training time is changed to produce enough output voltage and gripping force for the desired object.

Unlike conventional spike-based control algorithm methods through a central processor, the sensing data are processed by the neuromorphic system in an automatic response way, which is similar to the unconscious response without processing in the brain like human beings. Thus, a bioinspired artificially intelligent robot with a trainable reflex arc is demonstrated. The demonstrated neuromorphic grasping method mimics reflex arc. By using the proposed neuromorphic system, the energy consumption for the grasping can be reduced when it is compared with the conventionally processed motion of robot hands. Future works on the improvement of the motion of robot hands and preparation of more efficient synaptic transistors can make the neuromorphic humanoid hand reduce energy use and faster response, which can be closer to the reflex arc function.

Discussion

A sensory humanoid hand with a proposed origami pressure sensor can detect variable resistance and connect with a portable neuromorphic system integrated by a 3D oscillator and a synaptic transistor for the transmission and processing of the signals. The 3D structural neuromorphic robot hand is created to provide a perception function, including the detection, transmission, and processing of signals. For the fabrication of such a complicated 3D neuromorphic system, we introduced a multi-axis robot DIW method. The six-axis robot-based DIW method printed conductive traces on curvilinear surfaces such as vertical and tilted surfaces seamlessly. We demonstrated a unique DIW method for the fabrication of 3D conductive traces by optimizing parameters such as the speed of robot motion to improve the efficiency of the 3D printing process.

The neuromorphic humanoid hand can also control its motion unconsciously, as the artificially intelligent robot mimics the reflex arc performance. The robot hand is trained to replicate the learning progress of gripping a ball, similar to the gripping motion of human hands. Different gripping forces are monitored through the compressed depths to the soft ball with different steps, such as before training, during training, and after training. The neuromorphic humanoid hand demonstrates an increased gripping force sufficient to hold an unknown object. The force output from our neuromorphic system after training is controllable by modulating the initial training force. Because the non-volatile effect of the neuromorphic device is dependent on the bias applied on the gate electrode of synaptic transistors, it is possible to modulate by control of the amplitude and duration of training force. So, different training effect is possible to control for grasping of various unknown objects. When the system grasps flexible objects, smaller training force needs to produce weak grasping force by reducing the enhancement during the training process, while stronger training force requires to produce a higher grasping force for a rigid object. Therefore, an artificially intelligent robot hand is exhibited with the demonstration of an unconscious response through training.

Limitations of the study

Integration of the 3D-printed neuromorphic system with a humanoid hand is the first attempt for the realization of an artificially intelligent robot using non-volatile effect. So, this research still needs to find the way to improve the efficiency of training. Our demonstration here can be the first step toward combination of multiple input signals to mimic the biological signal perception functions in future.

STAR★Methods

Key resources table

REAGENT or RESOURCE	SOURCE	IDENTIFIER
Chemicals, peptides, and recombinant proteins
Urethane acrylates	Allnex	EBECRYL® 4833
2,2′- azobis (2-methylpropionitrile)	Sigma aldrich	CAS# 78-67-1
Luperox A75 benzoyl peroxide	Sigma aldrich	CAS# 94-36-0
Methacrylic acid	Sigma aldrich	CAS# 79-41-4
Silver flakes	Inframat® Advanced Materials LLC	Product # 47MR-10F
Software and Algorithms
LabVIEW	National Instruments	https://www.ni.com/en-ca/shop/labview.html

Resource availability

Lead contact

Further information and requests for reagents should be directed to and will be fulfilled by the lead contact, Woo Soo Kim (woosook@sfu.ca).

Materials availability

This study did not generate new unique reagents.

Method details

Preparation and characterization of humanoid hand

An FFF 3D printer with a direct-drive type (Tl-D3 pro, Tenlog, ltd) was used to prepare the architectured 3D origami fingers and potentiometric sensors of the humanoid robot hands. The filament for the 3D printer was a commercial Ninjaflex TPU85A (Fenner Drivers, Inc.). The origami structures were designed using Solidworks as a 3D CAD file; then, the file was converted to g-codes using the Cura slicer program (Ultimaker, Ltd.) for the 3D printing process. The nozzle size of the hotend was 0.2 mm. The print speed, infill, layer height, temperature, and width for the printing process were 30 mm/s, 0.1 mm, 223°C, and 0.2 mm, respectively. A commercial polylactic acid (Ulimaker, Ltd) filament was used for the sensor frames and palm, and the FFF 3D printer (with a bowden tube type (Ultimaker 3)) was utilized for printing. The g-code preparation was the same as described above. After printing, each part was carefully assembled and integrated into a commercial six-axis robot arm (Niryo One, Niryo Inc.). The finger motion was actuated and controlled by strings connected to DC motors.

The active layer of the pressure sensor for detecting the external force was fabricated using a DIW printer (SHOT mini 100Sx, Musashi Engineering, Inc.) at a speed of 1 mm/s, followed by curing at 80°C for 1 h. A highly resistive paste (3.8 × 10³ Ω m) was prepared for the active layer using the solution process. Urethane acrylates, 2,2′- azobis (2-methylpropionitrile), and Luperox A75 benzoyl peroxide were dissolved in methacrylic acid at a ratio of 75:3:3:19. Then, 40 wt% of silver flakes were added and mixed using a SpeedMixer DAC150.1 FVZ-K (FlackTek, Inc.). The viscosity of the silver paste was controlled by heating to 65°C.

Determining the angles of kresling origami

After fully folded, the Kresling origami has the same area for the identical folded shape, a hexagon, as shown in Figure S1A. Due to this geometrical feature, the minimum angle of α for triangular designs is determined to be 30°. For more details, the six vertexes near the α angle of triangles at the folded origami form the hexagon vertex. Moreover, any hexagons on the same circumscribed circle generate identical areas. Thus, the vertex near the α angle for any possible triangle for the origami can not be out of the circumscribed circle to generate the cylindrical shape. This feature indicates that the minimum α angle to generate the Kresling origami is 30°, the triangles with a vortex on the circumscribed circle. Moreover, the triangle with α = 30° and β = 40° of Kresling origami has the same shape with its α = 40° and β = 30°. Thus, only the β angle is selected as controllable parameters to investigate the rotating behavior of the origami. As the design criteria, α and β angles has been obtained from geometrical properties (C. Jianguo et al., 2015).

$$\beta ={\sin }^{-1}\left(\frac{d}{a}\right)$$ (Equation 3)

$$ah=d\left(d\cdot \cot \left(\alpha \right)+\sqrt{a^2-d^2}\right)$$ (Equation 4)

where the a, h, and d are the lengths around triangular patterns represented by Figure S1B.

3D printing of a portable neuromorphic system

The 3D oscillator substrate was fabricated using an FFF printer (Ultimaker S3, Ultimaker), whereas the conductive traces were fabricated using a six-axis robot DIW printing system (SHOT mini 100Sx and ML-808GX, Musashi Engineering, Inc.). The IR laser (5 W, 808 nm) (IRM808TA-5000FC + ADR-180A Shanghai Laser & Optics Century Co. Ltd.) was modulated using a function generator (SDG2042X, SIGLENT Technologies North America, Inc.). The printing route was controlled by an industrial six-axis robot (R-30iB, FANUC America Corporation) using simulation software (RoboDK, RoboDK Inc.). The printed circuit board (PCB) of the neuromorphic system was designed using the software Altium Designer, and was printed using a PCB printer (V-One, Voltera). The synaptic transistor was prepared using electrolytes with poly (ethylene oxide) (PEO) (0.16 g), lithium perchloride (LiClO₄) (0.02 g), and methanol (1.8 mL), followed by drop-casting on the gate of the synaptic transistors. The output of the oscillator was monitored using an oscilloscope (SDS1052DL, SIGLENT Technologies North America, Inc.). Simultaneously, three synaptic transistors and one portable neuromorphic system were characterized using a source meter (Keithley 2400, Keithley Instruments), and were monitored through a self-made program by LabView.

Training for sensory humanoid hand

One neuromorphic humanoid hand was prepared and connected to the 3D printed six-axis robot arm (Niryo One, Niryo). The motion of the arm was controlled using the Niryo One Studio software. Meanwhile, the gripping motion of the neuromorphic humanoid hand was controlled by the developed neuromorphic system.

Published: April 15, 2022

Data and code availability

All data reported in this paper will be shared by the lead contact upon request.