Electronic skin as wireless human-machine interfaces for robotic VR

The closed-loop HMI system could compliantly interface with human body for teleoperating various robotics with haptic feedback.

Abstract

The coronavirus pandemic has highlighted the importance of developing intelligent robotics to prevent infectious disease spread. Human-machine interfaces (HMIs) give a chance of interactions between users and robotics, which play a significant role in teleoperating robotics. Conventional HMIs are based on bulky, rigid, and expensive machines, which mainly focus on robots/machines control, but lack of adequate feedbacks to users, which limit their applications in conducting complicated tasks. Therefore, developing closed-loop HMIs with both accurate sensing and feedback functions is extremely important. Here, we present a closed-loop HMI system based on skin-integrated electronics, whose electronics compliantly interface with the whole body for wireless motion capturing and haptic feedback via Bluetooth, Wireless Fidelity (Wi-Fi), and Internet. The integration of visual and haptic VR via skin-integrated electronics together into a closed-loop HMI for robotic VR demonstrates great potentials in noncontact collection of bio samples, nursing infectious disease patients and many others.

INTRODUCTION

Human-machine interfaces (HMIs) in teleoperated robots have attracted great attention due to their great potential in many application areas, such as immersive games (1, 2), prosthesis or exoskeleton control (3–5), and extreme environment operation (6). Although substantial progress has been made in the development of HMI systems and equipment, most of them rely on cumbersome machines and rigid electronics that have constraints in terms of wearability, comfortability, and limited functionalities (7–9). With the fast development of flexible electronics, HMI system is switching from conventional bulky equipment to miniaturized and integrated alternatives on flexible substrates (10–12). Up to now, various flexible sensors have been developed to detect electrophysiological signals, such as electroencephalography (12), electrooculography (13), and electromyography (10, 14, 15) or capture human motion (16–19) and then convert this information into instructions for robotic control. Recent advances in engineering enable HMIs to realize wireless transmission function via near-field communication (20, 21), Bluetooth (22–24), and Wireless Fidelity (Wi-Fi) (14). However, the current flexible/wearable HMIs are only capable of acquiring information from parts of the body, such as hands or arm (table S1) (2, 11, 14, 18, 20, 22, 23, 25–38). Moreover, lack of essential feedback system for self-regulating behaviors limits the applications in conducting complicated tasks in dynamic environment. Therefore, the development of closed-loop (CL) wearable systemic HMIs is the key to improve this area for controlling advanced robotics. As the next generation of HMIs, the CL-HMIs should exhibit great advantages over the existing technologies as follows: (i) fully flexible configurations with skin-integrated elements so that they can accurately capture extensive body motional information as precious instructions, (ii) long-range wireless transmission function that enables transporting users’ instructions to manipulate the targeted equipment or machine at any corner of the world as long as covered by Internet, and (iii) well-designed feedback systems that allow users to synchronously receive enough information from the robots and make adjustments timely.

RESULTS AND DISCUSSION

This paper introduces a complete set of materials selection, device design, integration strategies, and system layouts for achieving a skin-integrated electronic system as a wirelessly controlled HMI with skin-like sensors and miniaturized soft actuators. The fully programmable CL-HMI system realizes broad application means of robotic virtual reality (VR) via interacting with robots in a combined visual, auditory, and haptic VR manner (Fig. 1A). Figure 1B shows a schematic diagram of the system, referred to as the skin-integrated CL-HMI. The CL-HMI adopts a layout of multilayer stacking, where a layer of skin-tone elastomeric silicon [~0.4 mm; poly(dimethylsiloxane) (PDMS), ~145 kPa] serves as the soft adhesive interface to the skin. The copper (Cu) traces (200 to 1000 μm in width, and 6 μm in thickness) supported by polyimide (PI; 12 μm) interconnect a collection of chip-scale integrated circuits and sensing components, including resistors, capacitors, Bluetooth module, microcontroller unit (MCU), bridges, and our self-developed soft sensors and actuators, and act as wireless power transmitting antenna. All Cu traces are designed as the filamentary serpentine structure based on well-established mechanics design rules, resulting in the stretchability of whole system. Another layer of skin-tone soft silicone (PDMS, ~145 kPa) encapsulates all functional components (fig. S1). Figures S2 and S3 show the photos and schematic illustration of the soft skin sensors where the functional component is a piezoresistive thin film on the top of interdigital electrodes. These sensors work as bending sensors, which can be mounted on different joints and accurately capture body motions via resistance change in different bending angles. In addition, the same sensors also function as the pressure sensor on the robots for detecting the robotic activities. The haptic feedback system adopts skin-integrated vibratory actuators based on Lorentz forces effect, and its vibration amplitude is controlled by frequency. In the actuator, a permanent magnet (nickel-plated neodymium magnet, diameter of 5 mm, and thickness of 0.5 mm) is glued on the center of a thin disk of polyethylene terephthalate (PET; 125 μm) with a semicircular slit and located on the top of a resin ring (inner and outer diameters of 5.7 and 7 mm, respectively, and thickness of 1.5 mm). The magnet vibrates when the pulse width modulation (PWM) current passes through the bottom coil (Cu coil, 50 μm of Cu wire diameter, and 41 turns) (fig. S2 and S3). The frequency of the PWM current to the Cu coil is adjustable according to the feedback information from controlled robots. Seven sensors (bending sensor) and five actuators are directly connected to the analog-to-digital converters (ADCs) and general purpose inputs/outputs (GP I/Os) of MCU in the CL-HMI patches for multiple channel sensing and actuation, respectively. Figure 1C presents the overall design of the circuits and system in the skin-integrated CL-HMI, where the sensors convert human motion into electrical signal, which is further processed by the MCU and wirelessly transmitted to the targeted robot. Meanwhile, contact with external environment detected by the pressure sensors on the robots serve as feedback signal to control the vibration intensity of the haptic actuators through the Bluetooth module, thus providing haptic feedback to the user (Fig. 1C and fig. S4). A rechargeable Li battery (3.7 V, 60 mA·hour, and 15 mm by 10 mm by 5.8 mm) is connected to the circuit with a pony-sized switch in series, which can support the skin-integrated CL-HMI with full-load operation over 68 min. In addition, the surrounding Cu coils (inductive reactance, 9.84 μH) in series with a 14-pF capacitor can realize wireless charging with the resonant frequency of 13.56 MHz.

Fig. 1. Design and architecture of the epidermal CL-HMI system.

(A) A schematic illustration of the concept of robotic VR, where a nurse is wearing the CL-HMI to teleoperate a robot for reading a thermometer with virtual and haptic feedback. (B) Exploded view schematic illustration of the CL-HMI connecting with seven bending sensors (BSs) and five actuators. (C) Circuit diagram of the CL-HMI system. (D) Design of the flexible circuits in the CL-HMI. (E) Optical images and schematic illustration of the circuit design with enlarged views inset. (F and G) Finite element modeling (F) and optical images (G) of the CL-HMI in twisted, stretched, and bended configurations. Photo credit: Yiming Liu, Department of Biomedical Engineering, City University of Hong Kong.

The overall size of the control panel of the skin-integrated CL-HMI is 57 mm by 39 mm by 0.8 mm (Fig. 1D). The control panel takes the form of a multilayer stack that includes a thin elastomeric layer as a soft, adhesive interface to the skin and a functional layer that supports the wireless control and feedback systems (Fig. 1E and figs. S5 and S6). The tiny buckled Cu wires serve as stretchable bridges to connect sensors, the battery, and other electronic components (fig. S6). Owing to the high adhesion of the elastomeric layer, the CL-HMI can be tightly mounted on various locations of the human body by van der Waals forces for long-term use without falling off, such as forearm, leg, back, and tummy (figs. S7 and S8). Figure S9 presents the adhesive duration as the whole CL-HMI system is mounted on a volunteer’s arm and under continuous bending from the user’s arm for 8 hours, where the results show that the control panel and five actuators can tightly interface with skin for the entire test duration without falling off from skin. While the adhesive duration for bending sensors ranges from 3.35 to 5.87 hours. These results demonstrate long-term stable operation of the system without the concern of falling off. The filamentary serpentine interconnects and the geometrical layouts of the functional components are guided by mechanical theory that offers excellent mechanical performance through the entire device. The circuit parts are located at the mechanical neutral plane between thin (0.4 mm thick) and soft (modulus, 145 kPa) top/bottom PDMS layers to minimize the applied strain/stress in the circuits during deformation. Figure 1F shows that the simulated equivalent strains in the copper layers remains lower than the elastic limit (0.3%) in ~90° twisted and ~180° bended (bending radius, ~2 cm) devices. In addition, finite element analysis (FEA) results show that the filamentary serpentine interconnects can buckle, twist, and deform in and out of the plane with little constraint from the soft encapsulation layer, proving that the structure decreases the strain/stress in the copper traces even with extreme stretching imposed on the entire device (Fig. 1F and fig. S10). The maximum equivalent strains in the copper layer are less than 3% in the device stretched by 20%, which is lower than the fracture strain (5%) of copper. Abovementioned excellent mechanical performance allows the entire device to be stretched, bent, and twisted for seamless, conformal integration onto the curvilinear surfaces of human epidermis (Fig. 1, F and G). Furthermore, the electromagnetic properties of the energy harvesting antenna undergo negligible changes when stretched by 20% and bent to a radius of ~2 cm, thus ensuring robust wireless power harvesting, even after extreme deformation (figs. S11 to S13).

Optimizations in materials selection, device layout, and mechanic design of the bending sensors offer sensitive operation as motion recognition and tactile sensation via bending and pressing of the piezoresistive sensors. Figure 2A shows the electrical response of the bending sensor at different bending angles from 0° to 150°, corresponding to the bending radius from ∞ to 3 mm. The relative resistance variation (ΔR/R₀) of the bending sensor increases from 0 to 5.92 with the bending angles’ increase from 0° to 150°, where R₀ (measured of 0.88 megohms) is the initial resistance without any mechanical deformations and ΔR is the resistance variations. Figure S14 (C and D) presents the stable electrical performance of the bending sensor bent with angle of 45° at a constant frequency of 1 Hz for ~1900 cycles by the self-developed bending platform (fig. 14, A and B). To evaluate the sensor performance at different bending speed, we measured resistance change with bending angle of 45° at four different frequencies, ranging from 0.5 to 4 Hz, and the results show that the bending sensor yields a constant response with different bending frequencies (fig. S14E). Figure S15 shows the electrical response of seven bending sensors mounted on a tester’s arm under both dry and wet skin conditions. The results indicate that the humidity of the skin surface has negligible effect on electrical performance of the bending sensors because of the waterproof property of the encapsulation layer (PDMS). The sensors attached on the joints offers obvious ΔR/R₀ differences, which allow precise and robust operations as motion recognitions in skin-integrated CL-HMI. Meanwhile, the sensors can also be integrated on the robots/machines to serve as the pressure sensors to collect tactile information. Figure 2B shows the electrical response of a representative sensor integrated on a robotic hand for detecting pressure, where it can be seen that the sensor exhibits excellent sensitivity and a linear relationship with applied mechanical pressure. The pressure sensor can accurately capture the mechanical pressure from 0 to 120.5 kPa with ΔR/R₀ variation from 0 to 0.72. Figure S16A shows the stable electrical response of the pressure sensor under a constant stress of 32.4 kPa at four different frequencies, ranging from 0.5 to 4 Hz, which proves the negligible effect of frequency on the electrical performance of the pressure sensor. Figure S16 (B and C) shows the electrical response of the pressure sensor under over 8500 cycles of 32.4-kPa mechanical stress loading at a frequency of 1 Hz. Only a slight fluctuation in amplitudes was observed after thousands of cycles of tests, which proved the stability and durability of the pressure sensor, indicating the possibility for long-term use. Figure S17 shows the response of voltage output in the control panel under multiple cycles bending at a constant angle of 75° and frequency of 1 Hz for 1 hour. The stable response without any signal interruption illustrates the stability of the control panel in potential practical applications. Various kinds of mechanoreceptors under skin incorporate with different tactile input types, i.e., pressing and vibration. Since frequencies in the range between 100 and 300 Hz provide the strongest tactile feelings to human skin, we design the vibratory actuators to realize resonant frequencies in this range that allows the actuator to operate efficiently with low power consumption and high output intensity. Systematic modeling of materials and mechanics for the actuators by FEA allows the actuators to resonate at the target frequency range. Figure 2C presents the simulated resonate frequencies of the actuators mounted on a glass (modulus, 55 GPa) and a tissue layer (modulus, 130 kPa), which change from 0 to 600 Hz as a function of the central angle (0° to 120°) of the PET handing layer. Guided by the FEA results, we choose the PET layer with 60° central angle for generating a resonate frequency of 250 Hz (Fig. 2C and movie S1). At an input power of 0.15 W, the highest vibration amplitude of the actuator can reach up to 0.3 mm at the resonate frequency, arousing the strongest haptic feeling (fig. S18). To offer various distinguishable haptic information, five vibration levels at different frequencies (20, 50, 83, 166, and 250 Hz) are selected to serve as haptic feedback from weak to intense.

Fig. 2. Electrical characteristics of the CL-HMI system.

(A) The electrical response of the bending sensor as a function of bending angles from 0° to 150°. (B) The resistance response of the pressure sensor at different mechanical pressure. (C) The FEA results of the resonate frequency of the actuators on a glass or skin tissue versus the central angle of the PET handing layer. (D) The largest communication distance between the control panel (Sender) in the CL-HMI and the Arduino board (Receiver) with an identical Bluetooth change versus the thickness of the top PDMS encapsulation layer. Here, the control panel was mounted on a human knee and measured in face-to-face mode to the receiver. (E) The ultimate transmission distance of the CL-HMI for operation on different locations, including the desk (A1), back hand (A2), forearm (A3), abdomen (A4), and knee (A5). (F) Response time as a function of transmission distance between sender and receiver at Wi-Fi mode. (G) The continuous working duration of different battery with different initial voltage. (H) The battery voltage decrease of the CL-HMI in a standby mode. (I) Wireless charging plots of the battery in the CL-HMI at a working frequency of 13.56 MHz and power of 41 W, where the distance between transmission antenna (outer diameter of 2 cm; number of turns of 10; resistance of 1 ohms) and the CL-HMI is 5 mm.

Figure 2D presents the largest wireless transmission distance of the skin-integrated CL-HMI under Bluetooth mode, which is affected by the thickness of the top PDMS encapsulation layer. The measurement was taken where the skin-integrated CL-HMI attached on a volunteer’s knee to ensure the two identical Bluetooth modules face to face. Optimization in thickness of the PDMS encapsulation layer (0.4 mm) guarantees that the CL-HMI maintains both considerable wireless transmission distance and robust mechanical properties. The transmission distance of the skin-integrated CL-HMI is also relevant to the locations worn on the human body. Figure 2E compares the transmission distance (the sender and receiver are face to face) differences for the skin-integrated CL-HMI attached on different locations, where the maximum distance was detected from the device put on a desk (A1, ~11.8 m). We further compare the results from skin-integrated CL-HMIs mounted on human body, including the back of the hand (A2), forearm (A3), abdomen (A4), and knee (A5), whose wireless transmission distance remain stable in the range from 2.72 to 3.85 m but shorter than that on the desk. The reason of the transmission distance difference is due to the absorption of radio frequency (RF) electromagnetic radiation by complicated bio-tissues in human body (39). In addition, facing the direction between the sender and receiver of the Bluetooth modules also influences the ultimate communication distance, where the face-to-face situation (B1) achieves the longest value of 3.42 m (fig. S19). Although the transmission distance is affected by many factors, it is still constant over several meters in all situations and meets the envisioned application scenarios.

To further demonstrate its performance of real-time wireless operation, we measured the response speed of a CL cycle of signal sensing–data transmission–signal receiving–actuation response over distance ranging from 1 to 5 m (fig. S20). The response time in all five situations are typically <4 μs, indicating the excellent performance in wireless manipulation of robots with negligible time delays. In addition to the Bluetooth operation mode, the skin-integrated CL-HMI also supports another two wireless communication modes of Wi-Fi and Internet. For the Wi-Fi mode, the wireless transmission range can be extended to hundreds of meters as long as the operation area is covered by the same Wi-Fi network. From Fig. 2F and fig. S21, the response time of the skin-integrated CL-HMI for operating in 1-cycle CL duration occurs less than 350 μs, as the distance between sender and receiver is <60 m under Wi-Fi mode. The response time is much shorter than a typical human reaction time (550 μs) of making a decision after receiving tactile stimuli (40). While for Internet operation mode, the wireless transmission distance can be greatly extended to anywhere covered by Internet, where the response time ranges from 30.2 to 47.8 ms, highly relying on the Internet environment (fig. S22). The power source of the skin-integrated CL-HMI relies on a small Li-ion battery whose operation duration is up to a couple of hours. Figure 2G shows the working duration of the skin-integrated CL-HMI as a function of the initial battery voltage without actuating the haptic feedback, where the longest operation duration can last 240 min, which is sufficient to conduct a series of complicated tasks. Under the full-load operating mode with five actuators working at a frequency of 250 Hz, the working duration of the skin-integrated CL-HMI is still over 68 min (the initial battery voltage of 4.2 V) (fig. S23). The standby time of the skin-integrated CL-HMI can last over 330 hours (Fig. 2H). The outer antenna enables power harvesting via electromagnetic induction effect, supporting the wireless charging function for the skin-integrated CL-HMI with international standard RF transmission antennas/readers at 13.56 MHz (Fig. 2I).

A representative application of the skin-integrated CL-HMI system is to wirelessly control a robot. Here, a self-developed mobile robot (remote-controlled car) is first demonstrated, as shown in fig. S24A. The skin-integrated CL-HMI in this demonstration consists of a control panel and two bending sensors attached on the user’s two index fingers, where the left one controls the speed and the right one controls the direction (fig. S24B). Figure S24 (C and D) presents the resistance response of the two sensors for giving four instructions, including turning right, turning left, moving forward, and moving backward. The user can teleoperate the speed and moving direction of the remote-controlled car via the motion of the two fingers at Bluetooth mode by the CL-HMI system mounted on his forearm (fig. S25 and movie S2). The greatest advantage of the CL-HMI system is more accurately interacting with robots with the assistance of adequate feedback information beyond the conventional visual feedbacks (Fig. 3A). To prove the superiority of our haptic feedback system, we demonstrate another representative application of the CL-HMI to control a 7-degree-of-freedom (DOF) prosthetic hand with five actuating channels, as shown in Fig. 3B, figs. S26 and S27, and movie S3. There are seven bending sensors, which are mounted on five finger joints, wrist, and elbow of the user, recognizing the arm motion and converting this information into electrical signals to wirelessly control the prosthetic hand with an Arduino board (fig. S26). The five pressure sensors integrated on the five robotic fingers are capable of accurately detecting the mechanical forces applied on the prosthetic hand when grasping an object. The detected pressure serves as the feedback signal that determines the vibration amplitude of actuators for providing different tactile sensations. Thus, the user can precisely control the force of the prosthetic hand depending on the intensity of the haptic feedbacks. Figure 3C presents the process of a user controlling the prosthetic hand to grasp a balloon via the skin-integrated CL-HMI. The grasping force of the prosthetic fingers, determined by the bending angles of the user’s fingers, can be precisely controlled on the basis of judging the feedback intensity (Fig. 3D and movie S4). To further highlight the significance of haptic feedback in CL-HMI, we conducted another demonstration associating with the recognition of the mechanical properties of alike rubber (2.5-cm cubes), whose Young’s modulus is different but hard to distinguish visually (Fig. 3E). The pressure sensors respond to the hardness of each cube when the 7-DOF prosthetic hand is grasping them (Fig. 3F), and the signals are transmitted to each actuator for giving corresponding vibration intensity (Fig. 3, G and H). On the basis of the haptic feedback, users are able to sequence the cubes on the basis of their hardness and get a better ability to control force when holding a load. To demonstrate the feasibility of the HMI system for more freedom sensing and control, a commercial accelerometer (BNO080) serving as the motion sensor in the HMI was used to detect the wrist motion to control a 7-DOF robotic arm (fig. S28). By accurately recording angle variations of the wrist, we could manipulate the robotic arm to bend down, up, left, and right and rotate without any delay.

Fig. 3. Demonstrations of the control and feedback function of CL-HMI in teleoperating robots.

(A) Schematic illustration of HMIs with traditional HMI and CL-HMI system. (B) Photos of a skin-integrated CL-HMI mounted on forearm for controlling a 7-DOF prosthetic hand and experiencing haptic feedbacks. (C and D) Demonstration of the prosthetic hand grasping a balloon with three distinguished forces and corresponding electrical signal of the bending sensors, pressure sensors, and actuators. (E) Five rubbery cubes with different Young’s modulus, which have been colored by pigments for distinguishing (pink, brown, green, blue, and yellow). (F) Optical images of the CL-HMI mounted on human hand for controlling the 7-DOF prosthetic hand to grasp cubes; the inset image is the electrical response of bending sensors. (G) Pressure sensors on the prosthetic hand detect the applied pressure under the same grasping force. (H) Actuators provide adjustable vibration amplitude based on the pressure signals. Photo credit: Yiming Liu, Department of Biomedical Engineering, City University of Hong Kong.

Combining the skin-integrated CL-HMI with visual information, this two-mode system creates a advanced technology, referred as intelligent robotic VR, that exhibits huge potential in remote control technology. Figure 4A and fig. S29 show the schematic illustrations of this concept, where a user wears four skin-integrated CL-HMI patches to control a 13-DOF humanoid robot as well as synchronously experience the tactile information from forearms, upper arms, thighs, thigh sides, tummy, and crura in the robot with corresponding pressure sensors. In each skin-integrated CL-HMI patch, the number of bending sensors and haptic actuators can be freely selected according to specific applications. The user wears a VR glass at the same time to synchronize the visual information obtained from the electronic eyes on the robot. Figure 4B shows the logic circuit design for the robotic VR system that includes both “control” and “feedback” modules. In the control system, resistance change of bending sensors is converted into voltage change by voltage divider circuits (Fig. 1C and fig. S30). Then, the voltage change is read by seven general GP I/O ports with 10-bit ADC on the microcontroller. Data from each 10-bit (2¹⁰ = 1024) ADC enable converting analog voltage inputs into digital outputs from “0” to “1023” as long as the input is no less than the maximum detectable voltage. Furthermore, the conversion from the digital outputs to a 2-byte number group enables rapid and efficient operation of the wireless system without time delay (fig. S31). A system having seven GP I/O ports with ADC forms a 14-byte message and sends it out via universal asynchronous receiver/transmitter to the Bluetooth module. The Bluetooth module converts the input data into Bluetooth formats automatically and then sends it to a paired Bluetooth module that is connected to a PC (sender). The sender converts and transmits the data to the targeted PC (receiver) (the PC that sends commands to the robots) by Internet. Next, the receiver analyzes the signal and sends instruction to the Arduino board for controlling the robot via Bluetooth (fig. S22 and movie S6). The feedback system follows the same operation logic and uses the same path to transmit 10-byte messages from five pressure sensors backward to control vibration frequency and amplitude of five actuators. Five different vibration frequency levels ranging from 20 to 250 Hz are set for reappearing different tactile patterns and pressure levels of the robots.

Fig. 4. Demonstrations of the CL-HMI in robotic VR applications.

(A) A man wearing CL-HMI electronic skin wirelessly manipulates a robot through Internet with haptic and visual feedback. (B) Diagram of the command interface that supports the cooperation of four control panels, 13 bending sensors, and 13 actuators. (C) Photographs of the robotic VR in the example of saliva sample collection by nasopharyngeal swab. PSs, pressure sensors. (D) Demonstration of robotic VR in the real-time interaction between the user and a 13-DOF humanoid, doing complicated tasks including crouching, room cleaning, and patient care. Photo credit: Yiming Liu, Department of Biomedical Engineering, City University of Hong Kong.

Coronavirus disease 2019 infected by SARS-CoV-2 has raised global panic for its highly contagious and pathogenic virus features (41). In this period, one of the most important applications of the robotic VR is assisting medical staff to conduct medical tasks that can both relieve the pressure of limited medical resources/manpower and decrease the risk of exposure to infectious environments of SARS-CoV-2. Figure 4C and movie S5 showcase a possibility of robotic VR in sample collection for SARS-CoV-2 detection. Here, a medical staff conducts nasopharyngeal swab for sample collection by teleoperating a 7-DOF robotic hand with the help of the real-time visual information and tactile feedbacks. In the example of Fig. 4D, a medical staff wears a VR glass and skin-integrated CL-HMI patches across different locations of the body. The user can remotely control an intelligent robot to conduct various complicated tasks, including squatting, walking, cleaning room, and nursing patients, via the robotic VR system in Wi-Fi mode (Fig. 4D, fig. S32, and movie S7).

The robotic VR system reported in this work exploits a new-generation thin, soft CL-HMI that can directly integrate on skin as a platform for remotely controlling robots in a wireless operation mode. This type of skin-integrated CL-HMI is significantly distinguished in formats and functions over the previous wearable HMI. Extensive experiments and theoretical studies on materials, devices, and integration schemes in the system build a solid foundation for the robotic VR development. The above-introduced examples of high-DOF robots in robotic teleoperation, prosthetic control/feedback, and medical applications are representative of its broad potential applications in different areas, including medical robotics, industrial manufacturing, entertainment experiences, and surgical training.

METHODS

Fabrication of piezoresistance-based bending and pressure sensor

PDMS (purchased from Dow Corning Corporation) is used as the encapsulation layers for the soft sensors. As shown in fig. S3A, the self-developed bending sensor consists of two encapsulation layers (PDMS) and a functional sensing layer. The functional sensing layer (0.28 mm thick) with interdigital in-plane electrodes in a thin-film format was purchased from Spectra Symbol Corporation. The functional layer was cut into 30 mm–by–7 mm sheets, which defines the overall dimension of the skin-integrated sensor. Cu (diameter of 125 μm) wires bonds on electrode patches of the sensor by conducting resin (Shenzhen Ausbond Co. Ltd) and connects with the CL-HMI control panel. Then, skin tone–colored PDMS (cross-link: PDMS, 1:30; elastic modulus, 145 kPa) that encapsulated the sensing layer prevents delamination of the bonding patches and damages of the entire sensor during operation.

Fabrication of the haptic actuators

For the haptic actuator, a disk-shaped permanent magnet (thickness of 0.5 mm, diameter of 5 mm, and weight 0.05 g) is attached to the cantilever area of a PET disk. The PET disk (thickness of 0.125 mm) was laser-cut into a circle with outer diameter of 7 mm and 60° semicircular slits in the center. A three-dimensional printed ultraviolet (UV)–curable resin ring (outer diameter of 7 mm, inner diameter of 5.7 mm, and thickness of 1.5 mm) was mounted onto the PET disk by UV cure glue (Shenzhen Tegu New Material Co. Ltd) and then solidified under UV light. Last, a Cu coil (outer diameter of 7 mm, inner diameter of 3 mm, thickness of 0.2 mm, and wire diameter of 30 μm) was placed onto the other side of the resin ring, which is fixed by the same UV cure glue and solidified under UV light.

Assembly of the epidermal control panel

The fabrication started on a quartz glass, which was first cleaned by acetone, alcohol, and deionized (DI) water sequentially. Sodium stearate aqueous solution was spin-coated on the glass sheet, serving as a thin sacrificial layer, and then dried under 100°C for 5 min for later releasing steps. Then, a thin PDMS film at 600 rpm for 10s (0.4 mm) was spin-coated and baked at 110°C for 5 min for curing to form the stretchable substrate of the CL-HMI. Figure S33 shows the thickness of PDMS layers as a function of spinning speed, indicating the good controllability of the thickness. To ensure enough adhesion strength between the Cu circuit layer and the PDMS substrate, another ultrathin PDMS film was spread over the cured PDMS substrate before attaching PI-supported Cu film on it. After smoothly attaching the PI (12 μm)–supported Cu (6 μm) sheet on the PDMS substrate, all components are cured at 110°C for 5 min. Then, Cu sheet were patterned by photolithography process. Positive photoresist (PR; AZ 4620, AZ Electronic Materials) was spin-coated at 3000 rpm for 30 s, soft baked on a hot plate at 110°C for 5 min, then exposed to UV light for 45 s, and lastly developed for 1 min in the developer (AZ 400K). After etching and obtaining Cu patterns, the residual PR was removed by acetone and rinsed by DI water. The PI film not covered by Cu pattern was removed by reactive ion etching (Oxford Plasma-Therm 790 RIE System), thus yielding stretchable Cu circuits in the desired geometries. Low-temperature soldering paste (LF999, KELL YSHUN Technology Co. Ltd.) was used to bond and electrically connect all the components, including the Bluetooth (WH-BLE103), the microcontroller (ATmega328p-mu), bridge wires (resistance, ~0 ohms), microswitch (MSS-22D18G2), capacitors (14 to 22 pF), resistors (0.8 megohms), crystal oscillator (16 MHz), bending sensors, and haptic actuators, to corresponding contact pads on the Cu/PI substrate. After integration of all components, PDMS (cross-link: PDMS = 1:30, 145 kPa, 0.4 mm thick) were poured onto the device followed by curing at 110°C for 5 min. The total weight of an encapsulated CL-HMI system is 14.25 g with five actuators (0.13 g each), seven bending sensors (0.3 g per sensor), and a control panel (11.5 g). The skin-integrated CL-HMI can be worn on the skin for extended periods in various physical activities because of the high adhesion strength of PDMS, and the PDMS is biocompatible without causing any irritation (fig. S7). Adhesion of the control panel could be well maintained by cleaning the contaminant on the encapsulation layers using scotch tapes, which allows repeated utilizations.

CL-HMI system operation

For 7-DOF prosthetic arm, the voltage outputs of seven bending sensors will be acquired by seven GP I/O on a programmable MCU (ATMEGA328P-MU, Microchip Technology Inc.) with ADC function. Ten-bit ADC can convert the voltage signal from bending sensor into a number ranging from 0 to 1023 and then further translate them into a 2-byte value by the MCU to avoid information loss. The data are sent to the paired receiver (Arduino Mega 2560, Arduino) mounted on prosthetic arm through a Bluetooth module (WH-BLE103, Jinan USR IOT Technology Limited). The receiver system translates the 2-byte data into PWM signal, which are used to control the corresponding sensors on the prosthetic arm. The operation of the receiver system, with five pressure sensors attached to the prosthetic hand’s fingertips, follows a similar method to the CL-HMI. Electrical signals of five pressure sensors are converted into 2-byte data and then sent to the CL-HMI for haptic feedback. Higher pressure detected by pressure sensors results in a higher PWM current frequency output to the corresponding actuators, inducing a stronger vibration. For controlling the telecar, the CL-HMI consists of two bending sensors (one for speed control and the other for direction control) and a control panel for data collection, analysis, and wireless communication with paired telecar. The operation mode of the CL-HMI is similar to that of prosthetic hand teleoperation. In the teleoperation of prosthetic hand by Internet, resistance variations of the bending sensors are captured by the control panel and wirelessly transmit to the PC sender via Bluetooth. Program running on senders sets up User Datagram Protocol (UDP) communication with PC receivers, which is used to communicate with the prosthetic hand. As the data are received, senders automatically send all received data to the receiver using UDP. The receiver is connected with a router with a public IP address through WLAN. Network address translation is set in the router to specify a unique port in the public IP address for the receiver, allowing it to directly receive data from the Internet using UDP. The data received by the receiver will be further sent to the prosthetic hand through Bluetooth and then translated to motor rotation angle by MCU to do the corresponding motions. The feedback data will return to the sender using the same method for realizing the vibratory feedback generated by actuators. In the 13-DOF humanoid robot teleoperation demonstration, the CL-HMI system consists of four control panels mounted on four positions of the human body, 13 bending sensors, and 13 actuators. To avoid signal cross-talk in the receiver system and decrease response time of the whole system, each control panel of the CL-HMI system is paired to unique Bluetooth module in the receiver system. As a result, there are four independently operated Bluetooth modules in the receiver system of the 13-DOF robot.

Characterization

The resistance data in Figs. 2 (A and B) and 3 (D and G) and figs. S14 to S16, S24, and S25 were collected by a data acquisition/multimeter system (DAQ6510, Keithley). A Keyence VW-6000 high-speed video camera was used to measure the vibration amplitude of the actuator, as shown in fig. S18. The voltage data in Fig. 2I were measured by a PL3516/P 16/35 with a constant sampling frequency of 1000 Hz. The wireless charging was supplied through a power amplifier (power, 41 W; Electronics & Innovation 2100L Linear Power Amplifier) connected to an AC waveform generator (frequency, 13.56 MHz; 3390 50MHz Arbitrary Waveform Generator, Keithley). The external pressure load was generated by an oscillator (Vibration Generator, Ocean Technology Co. Ltd). The bending load was applied through a linear motor (LinMot PS01-23x160-R) with controlled frequencies and bending angles.

Mechanical simulation

The FEA commercial software ABAQUS (Analysis User’s Manual 2016) was used to optimize the design of interconnects and the layouts of functional components and to study the corresponding mechanical characteristics. The objective was to decrease the strain level in copper interconnects under different typical loads, including stretching, bending, and twisting. The PDMS was modeled by hexahedron elements (C3D8R), while the thin copper (6 μm thick) and PI (12 μm thick) layers were modeled by shell elements (S4R). The minimal element size was one-fourth of the width of the interconnects (200 μm), which ensured the convergence and the accuracy of the simulation results. The elastic modulus (E) and Poisson’s ratio (ν) used in the analysis were E_Cu = 119 GPa and ν_Cu = 0.34 for copper; E_PI = 0.8 MPa and ν_PI = 0.5 for PI; and E_PDMS = 145 kPa and ν_PDMS = 0.5 for PDMS.

Electromagnetic simulation

Finite element method was used in the electromagnetic simulations to determine the inductance, Q factor, and the scattering parameters S11 of the energy harvesting antenna in undeformed and deformed states. The simulations were performed using the commercial software Ansys HFSS (Ansys HFSS 13 User’s Guide), where the lumped port was used to obtain the scattering parameters S11 and port impendence Z. An adaptive mesh (tetrahedron elements) together with a spherical surface (800 mm in radius) as the radiation boundary was adopted to ensure computational accuracy. The inductance (L) and Q factor (Q) (shown in figs. S11 and S12) were obtained from L = Im{Z}/(2πf) and Q = |Im{Z}/Re{Z}|, where Re{Z} and Im{Z} represent the real and imaginary part of the Z, respectively, and f represents the frequency. For 20% stretching of the energy harvesting antenna, the changes of the inductance, Q factor, and resonant frequency are less than 5% (see figs. S11 to S13). For bending radius R = ~2 cm, the inductance, Q factor, and resonant frequencies are approximately unchanged (see figs. S11 to S13).

Supplementary Materials

This PDF file includes:

Table S1

Figs. S1 to S33

Legends for movies S1 to S7

Other Supplementary Material for this manuscript includes the following:

Movies S1 to S7

View/request a protocol for this paper from Bio-protocol.