Abstract
The current crisis surrounding the COVID-19 pandemic demonstrates the amount of responsibility and the workload on our healthcare system and, above all, on the medical staff around the world. In this work, we propose a promising approach to overcome this problem using robot-assisted telediagnostics, which allows medical experts to examine patients from distance. The designed telediagnostic system consists of two robotic arms. Each robot is located at the doctor and patient sites. Such a system enables the doctor to have a direct conversation via telepresence and to examine patients through robot-assisted inspection (guided tactile and audiovisual contact). The proposed bilateral teleoperation system is redundant in terms of teleoperation control algorithms and visual feedback. Specifically, we implemented two main control modes: joint-based and displacement-based teleoperation. The joint-based mode was implemented due to its high transparency and ease of mapping between Leader and Follower whereas the displacement-based is highly flexible in terms of relative pose mapping and null-space control. Tracking tests between Leader and Follower were conducted on our system using both wired and wireless connections. Moreover, our system was tested by seven medical doctors in two experiments. User studies demonstrated the system's usability and it was successfully validated by the medical experts.
Keywords: Telemedicine, telediagnostics, medical robots, tactile robotic, Covid-19
I. Introduction
Telemedicine can be referred to as a modern healthcare technology that allows patient-doctor interaction without any physical contact. This technology was initially proposed as a potential opportunity for people living in rural and remote areas to have access to safe, high-quality, and essential healthcare services [1]. Recently, telemedicine has received particular attention, especially due to the COVID-19 pandemic [2]. Several studies revealed that infections among medical staff urgently recommend preventive measures to protect them and to contain the spread of viruses, such as those caused by cross-infection [3]. It was shown that doctors and medical staff represent a considerable vector and must be viewed as a relevant risk for the patients. The frequent interaction and work in high-risk departments, such as quarantine areas, or initial contact are named among the greatest risk factors [3].
Apart from the current pandemic, infections and the spread of multi-resistant germs are problems for which satisfactory solutions have not yet been provided. Vancomycin-resistant enterococci (VRE) and multi-resistant gram-negative bacteria (MRGN) in particular show a problematic increase [4]. Regardless of the genesis (corona, multi-resistant germs), the need for care in medical facilities increases disproportionately in isolated patients. Most of them are old and chronically ill people who require special care. The consequences for care and maintenance are serious, even without an undoubted lack of care. Affected patients must be isolated on the wards to protect other patients in order to prevent further spread.
Considering the above factors, the design and development of teleoperation systems that provide remote healthcare services is a promising direction to advance the knowledge frontiers and state-of-the-art in the field.
II. Related Works
A. Robot Assisted Telemedicine
The proposed approach is based on an assisted teleoperation system that is integrated into an easy-to-use user interface (e.g., semi-autonomous teleoperation1/supervised autonomy2) – following the concept shown in Fig. 1. The basic goal is to investigate tasks of the functional class that lies between pure teleoperation (involving the operator) and full autonomy (not involving the operator). All of these model-based approaches are based either on predictive simulations of the remote environment or on an instruction abstraction between the operator and the robot (e.g., [5] for automated visual input processing). In order to enable more natural teleoperation in the presence of high delays, the model-based teleoperation uses environment models that are generated by a-priori knowledge and updated during the manipulation. In [6] a network-based controller with visual feedback was proposed, which sends available sensor data to distributed computing nodes over a network so that the data can be calculated in parallel. The amount of transmitted data (multi-modality) requires new concepts for network use and methods to control the network traffic through an optimized teleoperation control (e.g., [7]).
Fig. 1.
Illustrative concept of the proposed robot-assisted telediagnostics system with scientific advances described herein that allows multiple remote telediagnostic tasks whilst protecting medical doctors and staff from potential infections and hazards (illustration on the left). Real-world applications are described in the Section IV and the real setup is shown in Fig. 6(b).
Early approaches for telepresence3 applications focus primarily on the stabilization of the coupled telepresence system and provide a compromise between stability and precise reflection of the contact forces on the Follower side. Important fundamental works on this topic are [8], [9]. In [10] different approaches using the internet are compared experimentally. Particularly noteworthy are the approaches of the wave-scattering transformation [8], wave prediction with energy control [11], digital data reconstruction filters [12], wave integrals [13], passivity-based adaptive control [14] and passivity control in the time domain [15] and finally virtual energy tanks [16] in tank-based teleoperation.
B. Telemedicine and Telediagnostics Before COVID-19
Telemedicine, as a generic term, describes an area of medicine in which medical care is not provided directly, but over a more or less greater distance, [17]. It was particularly important in the field of teleradiology in sparsely populated regions [18] and in intensive medical care Telemedicine therefore primarily focused on the virtual provision of medical competence if this is not available on-site or it is associated with an unacceptable risk [19]. For instance, in the area of teleconsultation [20] and telemonitoring or in the form of telepresence, which enables active interaction with the remote site (for example through telestrators or telemanipulators) [21], [22]. The field of telediagnosis in the sense of a direct telecontrolled doctor-patient interaction, on the other hand, is still relatively unexplored and limited to few applications. Indeed, most of the scarce applications concerned extreme situations, for instance, in military settings, monitoring in extreme environments (e.g., the ascension of Mount Everest) [23], or for monitoring children patients with cardiogenic shock using thermodynamic cameras [24].
In a different context, a robotic system for telediagnosis of breast cancer was first introduced in [25], [26]. Therein, the system explored a haptic input device with force feedback and a robotic manipulator on the remote site to examine the patient from distance using the internet.
Another robotic system for telediagnosis is described in [27]. The Akibot mobile robot is used for telepresence applications in regions with poor coverage of medical facilities. It allows teleconsultations and has individual integrated diagnostic sensors: an otoscope for visual inspection of the ear canals, a stethoscope for auscultation, and an ultrasound probe to represent the sensory equipment. In Akibot, however, the use of these devices is left to the patient, which in turn significantly hinders its applicability.
In [28] proposed a system for remote auscultation using a 3-DoF (Degree of Freedom) robotic manipulator which certainly constraints the doctor in terms of the range of motion and tasks.
C. Telemedicine and Telediagnostics After COVID-19
The health and social-economic crises stemming from the current COVID-19 pandemic demanded and still demands novel solutions from all fields [29]. Particularly, the urge for a general diagnosis of potentially infected patients and environments, led to the fast development of novel telediagnosis solutions. Among the most promising results, it is worth mentioning, image transmission [30], recording vital parameters [2], performing cardiac and lung ultrasound for a COVID patient [31], ultrasound scanning [32], patients digital triage [33], blood draw and injections [34]. Khan et al. report the use of robotic systems for ultrasound diagnosis in remote regions in Canada [35]. Overall, it is clear that telemedicine represents a potential approach for coping with the current COVID crisis [2]. Nonetheless, all the aforementioned results are still tailored to very specific applications. More close to our hardware framework, the promising ReMeDi system is introduced in [36] – mostly for remote echocardiography examinations. Despite its relevant application, the system is still a prototype with a non-commercialized robot, which still requires assistant medical personnel with the patient for initiating the commands of the remote robot [37]. The haptic force-feedback therein is also conveyed only to the doctor. The system also relies on a 2D-display, neglecting depth information for the doctor. In this work, we address the aforementioned open challenges and bring to light the details of the implementation for reproducibility. Further details about the novelties of our proposed system and approach are described in the next subsection.
D. Beyond the State of the Art
This work is motivated by the challenges in current telemedicine and telediagnosis systems and, mostly, by the pressing context and situation of the health care system due to the COVID-19 crisis. Particularly, the motivation for developing our system arose in response to an exceptional—and urgent—request from the BMBF4 to form an interdisciplinary consortium consisting of roboticists and medical experts and to find innovative solutions focused mostly on the protection of the front workers in the fight against Corona, i.e., medical personnel. An example is an exposure during routine medical check-ups due to the necessity of physical examination (e.g., temperature measurement and blood pressure). In this work, we address such a challenge from a scientific perspective by providing the robotics solutions to build a new paradigm of telemedicine based on remote physical interaction. In this context, our work goes beyond the presented state-of-the-art. This is achieved using the following contributions:
-
•
Telediagnostic system that allows a medical doctor to achieve three different tasks: auscultation, palpation, and ultrasound.
-
•
Redundant teleoperation system in terms of implemented control modes (i.e., joint-based and displacement-based modes) and visual feedback received from the patient site allowing the doctor to choose in a flexible manner the teleoperation mode based on their actual tasks.
-
•
Dual haptic teleoperation control mode to allow, in addition to audio-visual communication, remote bidirectional haptic interaction between the doctor and patient, whenever it is required during any diagnosis (e.g., limb movements and physical examination).
-
•
User studies and experimental trials with seven medical doctors (experts) and patient actors to demonstrate our system usability and acceptance.
III. Method & Implementation
A. Hardware of the Proposed Telediagnostic System
The proposed telemedicine system incorporates the above-mentioned aspects and integrates them into an (i) modular-adaptive system, which can be easily and quickly adapted to new tasks after the implementation of the basic structure. This basic structure offers the advantage of retrofitting to a (ii) mobile system, which can be integrated (partially) autonomously into the clinical process and is ultimately and explicitly designed for (iii) use in the context of the COVID pandemic and therefore takes into account hygienic aspects which explicitly reduces the risk of person-to-person transmissions. Lastly, the system serves (iv) to protect the people involved in medical care.
Fig. 2 shows our proposed telemedical system in terms of hardware and software architecture.
Fig. 2.
Software and hardware description and control architecture for coupling the “Leader” and the “Follower” in the proposed telemedicine system. (a) Operator Teleoperation Joystick to move Leader end effector. (b) The doctor's site consists of a 7-DoF Franka Emika robotic arm Leader, desktop 2D screens, and one 3D screen. (c) The patient site consists of Follower 7-DoF Franka Emika robotic arm Follower, ceiling camera, and 3D stereo camera for video rendering and feedback. (d) Follower end effector consists of three different segments to conduct auscultation, palpation, and ultrasound. (e) Bilateral teleoperation interface concept. (f) Software and controller communication architecture of the telemedicine setup.
1). Doctor Site
Consisting of a 7-DoF Franka Emika robotic system.5 In addition, a joystick, conceptualized and implemented by Franka Emika5, was mounted at the robot end-effector to allow for a more intuitive interaction when manipulating the Leader robot arm. The latter allows the medical doctor to send positions and receive force feedback rendered at 1 kHz via the Leader robotic arm. Additionally, the doctor's site is equipped with audio-visual equipment used for both communication and audiovisual feedback rendering (i.e., heart beating and video-feed from the patient site).
2). Patient Site
Consists of a 7-DoF Franka Emika5 robotic arm Follower to interact with the patient. Moreover, this site is equipped with peripheral devices such as 2D camera (Panasonic AW-HN38H, 3D camera (z-mini) and medical measuring devices that are at disposal of the patient to collect - either manually or automatically - physiological data during the diagnosis process. The latter process is becoming easier due to the latest boost witnessed in terms of available wireless devices for remote diagnosis and monitoring due to the current pandemic. Importantly, a customized end-effector conceptualized and implemented by Franka Emika5 was mounted at the Follower end-effector to help the medical doctor conduct the auscultation, palpation, and ultrasound that are required for medical diagnosis. This was based on the feedback received from medical experts and collaborators collected during the requirement study on telemedicine system [38], [39].6
3). Control Loop and Communication
All devices have one common feature that makes their data (signals) and interfaces accessible via Ethernet network using the Robot Operating System (ROS). Thus, the ROS middleware is used in the background as a solution for inter-process communication in the back-end. For the teleoperation scenario, the Leader and Follower robots are both controlled together at a rate of 1 kHz. This can be achieved via the research interface (FCI) provided by Franka Emika5 which gives the possibility to connect the implemented controllers to the robots via the Ethernet network. In addition, both Leader and Follower robots can be controlled from one central controller thanks to franka_ros.7 Thus, the states of both robots are updated and the operator's commands are sent in real-time.
B. Visualization
Visual feedback is another crucial component of telepresence. For this purpose, we have integrated a 3D screen display from Seefront_Technologies (Fig. 2) in the doctor site, additionally to desktop screens. Moreover, the implemented 3D screen renders realistic 3D video images without using any special 3D glasses. This is achieved using embedded cameras that track both eyes' pupils. It has been shown that 3D stereoscopic displays enhance the user performance compared to 2D screens [40].
C. Teleoperation Algorithms
Our teleoperation setup consists of two identical 7-DoF robotic manipulators. Both robots are equipped with torque sensors and controlled via a torque interface that enables not only safe compliant physical human-robot interaction but also allows the evaluation of various dynamic behaviors across a large workspace. The operator (doctor) uses the “Leader” as an input device to control the “Follower” robot which interacts with the environment (patient). In addition, the robots are equipped with customized end-effectors to facilitate these tasks: the Leader features a joystick-like control handle with several buttons whereas the Follower is fitted with a multi-instrument medical end-effector. The dynamics of both systems are given by
![]() |
where denotes the inertia matrix and
and
are torques compensating Coriolis and gravity force.
are the torques applied by the control, the operator and the environment respectively. The subscript
and
refer to the Leader and Follower system. Finally, the joint position vector and its derivatives in time are defined as
. The two systems have a local network connection to a single central control computer. We implemented control algorithms to evaluate different bidirectional haptic control schemes regarding their utility performing various diagnostic procedures, which will be illustrated in the following sections.
1). Joint-Based Mapping
In the joint-space teleoperation mode, the two robots' position, velocity and applied torque are mirrored as closely as possible – also known as joint-to-joint – as seen in Fig. 3(a). The result is a highly transparent form of haptic teleoperation, as the systems posses identical kinematic as well as dynamic properties. To achieve this, the robots' states are first synchronized using a motion generator and then are coupled using teleoperation. The desired joint velocity and position
of the Follower system are simply computed based on the state of the Leader as
![]() |
where the gain is introduced to compensate for drift in position. The control torques are then set such that the Leader renders the environment torque
sensed at the Follower, while the latter uses a PD controller to track the desired signals,
![]() |
The design matrices are used to scale the applied torque and dampen the velocity to stabilize the system, whereas
define the gains of the PD-controller. The joint-based control scheme uses torques estimates, red via torque sensors at each joint of the robot, from the Follower as haptic feedback by applying the 7D torque vector to the Leader as formulated in (5). From the Follower side, the force feedback is applied using (6) based on the joint states of the Leader as given by the (3).
Fig. 3.
Results of tracking performance in joint-based teleoperation mode for Leader and the Follower in joint-to-joint mirroring. (a) Joint positions. (b) Force feedback mapping.
2). Displacement-Based Mapping
In the displacement-based algorithm, the Cartesian motion of the Leader w.r.t the initial poses is mapped on the Follower robot – as shown in Fig. 4. This control scheme allows more flexibility and intuitive control since the two robots do not need to be in the same configuration and arbitrary transforms might be applied to the mapping between the systems. Additionally, it allows the operator to disengage the teleoperation mode and reposition the Leader in a more appropriate configuration. This flexibility however comes with a loss of transparency as the configuration and therefore dynamic properties of the robots are no longer aligned. The error between the linear and angular
displacements is computed as
![]() |
where ,
denote the Cartesian end-effector position and orientation in quaternion representation respectively and the index 0 is introduced to denote the initial value. The error in pose for the Follower can then be written as
![]() |
with mapping the quaternion representation of the orientation to an angle-axis vector. Given the pose error, control laws rendering the wrench
sensed at the Follower on the Leader and a Cartesian PD-controller to minimize the pose error are formulated as
![]() |
Here, is the Jacobian and
are Cartesian PD gains. The vectors
are the robots' end-effecor pose and its time derivative. As in (5), the matrices
and
are properly chosen to scale the force and stabilize the system. Finally, the displacement-based control scheme uses the joints' torque estimates but projects them onto a 6D wrench vector in the end-effector frame using the Jacobian matrix. The wrench is then applied to the Leader as seen in (10) in the manuscript.
Fig. 4.
Results of tracking performance in displacement-based teleoperation mode for Leader and Follower. (a) Cartesian position. (b) orientation. (c) Force feedback mapping.
D. Tracking Performance
1). Joint-Based
In this experimental test, we conducted joint position tracking for all joints between the Leader and the Follower by moving the Leader in free space and so exciting all joints movements of both robots. In the second experiment, we conducted force feedback contact mapping and rendering by contacting a soft object (i.e., a sponge placed on the patient bed) using a palpation probe mounted at the end-effector. In both experiments, the robots are connected to the same Ethernet switch. The position tracking, as well as force mapping of both the Leader and the Follower, are illustrated in Fig. 3(a), Fig. 3(b). The results of this experiment show a delay of 1 ms for joint position/force tracking.
2). Displacement-Based
In this experiment, the end-effector position tracking between the Leader and Follower is conducted, by moving the Leader end-effector in free space. As in the previous experiment, the performance of force feedback mapping is evaluated using the palpation probe in contact with a soft sponge. The positions tracking and force mapping for both Leader and Follower are depicted in Fig. 4(a), Fig. 4(b), and Fig. 4(c). The results revealed 5 ms of delay between the Leader and the Follower in task space motion and force tracking.
E. Communication Delay Performance Analysis
In order to study the performance of the system under communication delay, the Leader robot is connected to the Follower via different wireless networks (wifi and mobile LTE or 4G). This imposes an average RTT latency of around 80 ms and 150 ms for wifi and LTE networks respectively. For the sake of comparison, the controller of the system is considered as in the previous experiment. The results are illustrated in Fig. 5(a)–5(d). Considering our position-force teleoperation scheme, the performance of the system is attenuated dramatically. The stability of the system is jeopardized, specifically during the interaction phases, which is felt by extra oscillation on the Follower sides as illustrated in Fig. 5(d). These performance issues can be alleviated using a faster network connection (e.g., 5G) or by implementing time delay compensation controllers to cope with latencies.
Fig. 5.
Results of communication performance between Leader and Follower over wifi and LTE network. (a) Joint positions in wifi configuration. (b) Force feedback mapping in wifi configuration. (c) Joint positions in LTE configuration. (d) Force feedback mapping in LTE configuration.
IV. User Experiments
To test and evaluate the performance of the system, two main experimental studies with medical experts were conducted within two different main tasks detailed below. In these two experimental studies, we used the joint-based algorithm and wired connection between Leader and Follower (since wired connection showed better tracking performance as reported previously).
A. Experiment 1: Familiarization
In this experiment, we asked seven surgeons (age: Mean standard deviation = 31
5 yo) to navigate within a maze-like structure using the Follower robot controlled via the Leader. Specifically, they were asked to navigate between four locations and perform four different tasks (Fig. 6(a)). At the first location (”i”) they were asked to perform a stethoscope-like task by making contact at this location, then move to the second location (”ii”) and navigate within the maze-like structure toward the third location. At the latter, they were asked to perform a palpation task (simulated abdominal area) by applying pressure to the three different tubes (”iii”). At the last location (”v”), they were asked to apply pressure as well at a precise location below the abdomen simulated area visited at location “iii”. Each time participants touched the maze-like borders via the Follower end-effector, the electric circuit is closed, thus their error score was incremented by one. Participants performed the experiment under two different conditions, namely, different orientations of the maze-like structure: horizontal (as a patient lying in bed) and vertical (patient standing in front of the robot) both facing the main camera viewpoint. By doing so we aim to investigate doctors' performance in both configurations in terms of perceiving depth, especially in the horizontal configuration. Each participant's doctor conducted three trials per condition. Conditions order was randomized across participants to control for any time effects (like learning, fatigue, de-motivation,..., etc.). We measured participants' performance in terms of task completion time and navigation error. Results of this experiment revealed that it took participants 153
56 sec (mean
standard deviation) to accomplish their 4 different tasks in the horizontal condition and 150
65 for the vertical configuration (Fig. 6(c)). In terms of errors, the experimental results revealed that participants touched the borders of the maze-like structure on average of 9
3 times and 7
4 times in both horizontal and vertical conditions respectively (Fig. 6(d)). Finally, likelihood-ratio test revealed no significant effect on the condition (vertical vs. horizontal) and its order for both dependent variables (i.e., time and number of errors;
in all cases).
Fig. 6.
Experimental setups and results. (a) Experimental setup of experiment 1. Yellow dots are stop points required in order to achieve the experimental tasks between four different locations. (b) Experiment 2 with representative doctor and patient actor. (b1) Doctor manipulating the Leader to achieve auscultation task viewed through 3D screen (b2) Patient lying in bed while the robot Follower is on mapping the Leader pose and forces. (b3) Palpation. (b4) Ultrasound. (b5) Ultrasound output images. (c) Completion time of experiment 1. (d) Number of errors committed in experiment 1. (e) Post-experiment 2 questionnaire results for doctors. (f) Post-experiment 2 questionnaire results for patient actors.
B. Experiment 2: Full Telediagnostic Examination
In these experiments, participants (7 surgeons) performed a full telediagnostic examination using the proposed system, through the order: audio-visual communication, auscultation, palpation, and finishing with ultrasound checkup. Seven actors (age: 32 11 yo, students at TUM) were asked to mimic and play the role of patients with a variety of abdominal symptoms for 20 cases (e.g. appendicitis or acute cholecystitis). After the experiment, participating doctors, as well as patients, were asked to fill in a post-questionnaire evaluating their experience and acceptance of the telediagnostic system. Specifically, doctors participants were asked to rate the setup regarding the level of “safety: did you feel safe when interacting with the system (robot arm)”, “intuitiveness: is it intuitive to use”, “communication quality: how was the audio-visual communication”, “Human in the loop involvement: do you prefer to be controlling the robot or the robot does it automatically” and “camera point of view: was it well located to achieve your task?” on a Likert scale from 1–5 (1 being very bad and 5 being very good). Patient actors were asked to rate the setup regarding the level of “safety”, “intuitiveness”, “communication quality”, “Usability of such a system”, “Fear of robot replacing a doctor” and “Necessity of presence of medical assistant nearby the patient cabin” on a Likert scale from 1–5 (1 being bad and 5 being good). Each experimental session of telediagnostic examination took between 15–20 min. Each participant doctor conducted three trials of the telediagnostic scenario on three different patient actors with different abdominal symptoms. Results of this experiment revealed that doctors successfully examined the patients and positively credited the system. All questionnaires (shown in Fig. 6(e)) variables were highly rated (between 4 and 5) with the exception of a few outliers. Further comments and feedback were also gathered from the doctor's participants where some expressed their opinion to further improve the system (e.g., “Robot arm must be trained in advance,” “Adding more cameras in the patient site,” With such a system I can do all the initial examinations that I usually do manually in my practice) From a patient side, results are reported in Fig. 6(f). The results highlight a similar acceptance and positive perception regarding the proposed system. Also, an interesting outcome is the correct understanding regarding the system object to support, and not replace, the health care personnel. These results indicate our system comes close to and could be comparable to a traditional examination in accordance with the basal requirements.
V. Discussion
In this work, we presented a telemedicine system to respond to medical doctor requirements for potential solutions during the current pandemics and infectious disease situations. Our system was tested in different network connections: wired, wifi and LTE. Moreover, our system was tested and experimented with by 7 medical doctors in two different experiments to evaluate their performance, system usability, and acceptance. Overall, experimental results showed that medical doctors credited our proposed system and it could be comparable to a traditional examination in accordance with the given telediagnostic requirements.
Our proposed telemedicine system allows for a complete routine medical check-up: from collecting physiological data (e.g., temperature measurement and blood pressure) to ventral and dorsal auscultation, tapping and abdominal palpation, and ultrasound scans. By using our system, medical doctors are enabled to interact with a patient, located in a different room, using audiovisual devices and a 7-DoF tactile robotic arm to bring the patient as close as possible to the doctors (as being in the same room) while protecting them from potential viral infections. For the haptic feedback rendering part, we developed the control algorithms within framework conditions for bilateral teleoperation.
The proposed system was tested with three different types of network connections: wired, wifi and LTE. In the wired configuration, we recorded a slight delay of 1 ms and 5 ms for joint-based and displacement-based modalities respectively. In wireless, we recorded higher delay between the systems: 80 ms and 150 ms for wifi and LTE network, respectively.
Finally, our proposed telemedicine framework can go beyond medical robotics for contagious diseases where it can be a potential solution and opportunity for people living in rural and remote areas to have access to safe, high-quality, and essential healthcare services. Especially within those areas with a lack of medical experts and specialists that are usually concentrated in the metropolitan areas.
VI. Conclusion & Future Work
In this work, we managed to conduct clinical trials with medical doctors at the hospital. Our experimental tests with the medical experts showed the usability and intuitiveness of the proposed telemedicine system use. More clinical trials will be our focus in near future work, especially, comparing different experimental variables. Certainly, this will concertize the proposed telemedicine system and will speed up the final approval for integrating it into hospitals.
Acknowledgment
We would like to thank Tamara Loibl, Clemens Kössler, and Benjamin Loinger for their support with the design and development of telemedical equipment and documentation as well as Viktor Korhammer and Sebastian Getz for supporting the electronics design and fabrication. Additionally, we would like to thank our Dr.med. Günter Steinebach, Matthias Richter, Melanie Porzenheim and Paul Thiel for their assistance during the submission. Author Sami Haddadin has a potential conflict of interest as a shareholder of Franka Emika GmbH, Munich, Germany.
Funding Statement
This work was supported in part by the German Federal Ministry for Education and Research (BMBF) under Grant 16SV8569 (ProteCT), in part by the Lighthouse Initiative Geriatronics by StMWiBayern through Project X, under Grants IUK-1807-0007 and IUK582/001, in part by LongLeif GaPa gGmbH through Project Y, in part by the Bavarian Institute for Digital Transformation (bidt-Project Responsible Robotics RR-AI), and in part by the German Research Foundation (DFG) as part of Germany's Excellence Strategy -EXC 2050/1 under Grant 390696704 – Cluster of Excellence Centre for Tactile Internet with Human-in-the-Loop (CeTI) of Technische Universität Dresden.
Footnotes
Allowing robots to perform duties that are otherwise hazardous to humans, while still giving the robot operators full control over specific tasks.
A system which is composed of a heterogeneous leader and follower robots, where the follower robot is assumed to be a redundant manipulator.
The telepresence concept can be defined as being present in a remote location virtually or via robotic avatars.
Federal Ministry of Education and Research–https://www.bmbf.de
[Online]. Available: https://franka.de
The selected telediagnostic tasks were defined based on the results of preliminary studies with 9 medical doctors and assistant personnel. They were asked about examination techniques as well as the extent of diagnostic categories necessary for a proper diagnostic telemedical system.
[Online]. Available: https://github.com/frankaemika/franka_ros
Contributor Information
Abdeldjallil Naceri, Email: djallil.naceri@tum.de.
Jean Elsner, Email: jean.elsner@tum.de.
Mario Tröbinger, Email: mario.troebinger@gmail.com.
Hamid Sadeghian, Email: hamid.sadeghian@tum.de.
Lars Johannsmeier, Email: lars.johannsmeier@tum.de.
Florian Voigt, Email: florian.voigt@tum.de.
Xiao Chen, Email: xiaoyu.chen@tum.de.
Daniela Macari, Email: daniela.macari@franka.de.
Christoph Jähne, Email: christoph.jaehne@franka.de.
Maximilian Berlet, Email: maximilian.berlet@tum.de.
Jonas Fuchtmann, Email: jonas.fuchtmann@tum.de.
Luis Figueredo, Email: Figueredo,figueredo@ieee.org.
Hubertus Feußner, Email: feussner@chir.med.tu-muenchen.de.
Dirk Wilhelm, Email: dirk.wilhelm@tum.de.
Sami Haddadin, Email: haddadin@tum.de.
References
- [1].Dolea C., Increasing Access to Health Workers in Remote and Rural Areas Through Improved Retention. Geneva, Switzerland: World Health Organization, 2009. [PubMed] [Google Scholar]
- [2].Khan Z. H., Siddique A., and Lee C. W., “Robotics utilization for healthcare digitization in global COVID-19 management,” Int. J. Environ. Res. Public Health, vol. 17, no. 11, pp. 38–19, 2020. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [3].Chang D., Xu H., Rebaza A., Sharma L., and Cruz C., “Protecting health-care workers from subclinical coronavirus infection,” Lancet Respir. Med., vol. 8, no. 3, 2020, Art. no. e13. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [4].Maechler F., Diaz L. A. P., Schröder C., Geffers C., Behnke M., and Gastmeier P., “Erratum to prevalence of carbapenem-resistant organisms and other Gram-negative MDRO in German ICUs: First results from the national nosocomial infection surveillance system (KISS),” Infection, vol. 43, no. 2, Apr. 2015, Art. no. 257. [DOI] [PubMed] [Google Scholar]
- [5].Probst T., Fossati A., and Gool L. Van, “Automated visual input analysis for a telediagnostic robot system,” in Proc. 5th Joint Workshop New Technol. Comput./Robot Assist. Surg., Katholieke Universiteit Leuven, 2015, pp. 54–57. [Google Scholar]
- [6].Wu H., Lou L., Chen C.-C., Hirche S., and Kuhnlenz K., “Cloud-based networked visual servo control,” IEEE Trans. Ind. Electron., vol. 60, no. 2, pp. 554–566, Feb. 2013. [Google Scholar]
- [7].Garcia E. and Antsaklis P. J., “Model-based event-triggered control for systems with quantization and time-varying network delays,” IEEE Trans. Autom. Control, vol. 58, no. 2, pp. 422–434, Feb. 2013. [Google Scholar]
- [8].Niemeyer G. and Slotine J.-J., “Stable adaptive teleoperation,” IEEE J. Ocean. Eng., vol. 16, no. 1, pp. 152–162, Jan. 1991. [Google Scholar]
- [9].Lawrence D., “Stability and transparency in bilateral teleoperation,” IEEE Trans. Robot. Automat., vol. 9, no. 5, pp. 624–637, Oct. 1993. [Google Scholar]
- [10].Rodriguez-Seda E. J., Lee D., and Spong M. W., “Experimental comparison study of control architectures for bilateral teleoperators,” IEEE Trans. Robot., vol. 25, no. 6, pp. 1304–1318, Dec. 2009. [Google Scholar]
- [11].Munir S. and Book W., “Internet-based teleoperation using wave variables with prediction,” IEEE/ASME Trans. Mechatronics, vol. 7, no. 2, pp. 124–133, Jun. 2002. [Google Scholar]
- [12].Berestesky P., Chopra N., and Spong M., “Discrete time passivity in bilateral teleoperation over the internet,” in Proc. IEEE Int. Conf. Robot. Automat., 2004, pp. 4557–4564. [Google Scholar]
- [13].Niemeyer G. and Slotine J.-J. E., “Telemanipulation with time delays,” Int. J. Robot. Res., vol. 23, no. 9, pp. 873–890, 2004. [Google Scholar]
- [14].Chopra N. and Spong M., “Adaptive coordination control of bilateral teleoperators with time delay,” in Proc. IEEE Conf. Decis. Control, 2004, pp. 4540–4547. [Google Scholar]
- [15].Artigas J., Vilanova J., Preusche C., and Hirzinger G., “Time domain passivity control-based telepresence with time delay,” in Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst., 2006, pp. 4205–4210. [Google Scholar]
- [16].Ferraguti F. et al. , “An energy tank-based interactive control architecture for autonomous and teleoperated robotic surgery,” IEEE Trans. Robot., vol. 31, no. 5, pp. 1073–1088, Oct. 2015. [Google Scholar]
- [17].Evans C. R., Medina M. G., and Dwyer A. M., “Telemedicine and telerobotics: From science fiction to reality,” Updates Surg., vol. 70, pp. 357–362, 2018. [DOI] [PubMed] [Google Scholar]
- [18].Rosenkrantz A. B., Hanna T. N., Steenburg S. D., Tarrant M. J., Pyatt R. S., and Friedberg E. B., “The current state of teleradiology across the United States: A national survey of radiologists' habits, attitudes, and perceptions on teleradiology practice,” J. Amer. College Radiol., vol. 16, pp. 1677–1687, 2019. [DOI] [PubMed] [Google Scholar]
- [19].Strotbaum V., Die Intensivmedizinischen Herausforderungen Der Zukunft Mithilfe Der Telemedizin Bewältigen. Wiesbaden, Germany: Springer, 2017. [Google Scholar]
- [20].Laurent-Bellue A., Poullier E., Prevot S., and Guettier C., “Le télédiagnostic extemporané,” Annales de Pathologie, vol. 39, no. 2, pp. 113–118, 2019. [DOI] [PubMed] [Google Scholar]
- [21].Erridge S., Yeung D. K. T., Patel H. R. H., and Purkayastha S., “Telementoring of surgeons: A systematic review,” Surg. Innov., vol. 26, no. 1, pp. 95–111, 2019. [DOI] [PubMed] [Google Scholar]
- [22].Schneider A., Wilhelm D., Bohn U., Wichert A., and Feussner H., “An evaluation of a surgical telepresence system for an intrahospital local area network,” J. Telemed. Telecare, vol. 11, no. 8, pp. 408–413, 2005. [DOI] [PubMed] [Google Scholar]
- [23].Harnett B., Satava R., Angood P., Merriam N., Doarn C., and Merrell R., “The benefits of integrating internet technology with standard communications for telemedicine in extreme environments,” Aviation, Space, Environ. Med., vol. 72, no. 12, pp. 1132–1137, 2001. [PubMed] [Google Scholar]
- [24].Nagori A., Dhingra L. S., Bhatnagar A., Lodha R., and Sethi T., “Predicting hemodynamic shock from thermal images using machine learning,” Sci. Rep., vol. 9, no. 1, pp. 1–8, 2019. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [25].Shen Y. et al. , “Supermedia interface for internet based tele-diagnostics of breast pathology,” in Proc IEEE/RAS-EMBS Int. Conf. Biomed. Robot. Biomechatronics, 2006, pp. 787–792. [Google Scholar]
- [26].Shen Y. et al. , “Internet based tele-diagnostic interface for breast pathology,” in Proc. 3rd IASTED Int. Conf. Telehealth, 2007, pp. 130–135. [Google Scholar]
- [27].Carranza K. A. R. et al. , “Akibot: A telepresence robot for medical teleconsultation,” in Proc. IEEE Intern. Conf. Humanoid, Nanotechnol., Inf. Technol., Commun. Control, Environ. Manage., 2018, pp. 1–4. [Google Scholar]
- [28].Falleni S., Filippeschi A., Ruffaldi E., and Avizzano C. A., “Teleoperated multimodal robotic interface for telemedicine: A case study on remote auscultation,” in Proc. IEEE Int. Symp. Robot Hum. Interactive Commun., 2017, pp. 476–482. [Google Scholar]
- [29].Bhaskar S. et al. , “Designing futuristic telemedicine using artificial intelligence and robotics in the COVID-19 era,” Front. Public Health, vol. 8, 2020, Art. no. 556789. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [30].Machado R. A., de Souza N. L., Oliveira R. M., Júnior H. Martelli, and Bonan P. R. F., “Social media and telemedicine for oral diagnosis and counselling in the COVID-19 era,” Oral Oncol., vol. 105, 2020, Art. no. 104685. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [31].Wang J. et al. , “Application of a robotic tele-echography system for COVID-19 pneumonia,” J. Ultrasound Med., vol. 40, no. 2, pp. 385–390, 2021. [DOI] [PubMed] [Google Scholar]
- [32].Akbari M. et al. , “Robotic ultrasound scanning with real-time image-based force adjustment: Quick response for enabling physical distancing during the COVID-19 pandemic,” Front. Robot. AI, vol. 8, pp. 1–14, 2021. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [33].Lai L. et al. , “Digital triage: Novel strategies for population health management in response to the COVID-19 pandemic,” Healthcare, vol. 8, no. 4, 2020, Art. no. 100493. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [34].Zemmar A., Lozano A. M., and Nelson B. J., “The rise of robots in surgical environments during COVID-19,” Nature Mach. Intell., vol. 2, pp. 566–572, 2020. [Google Scholar]
- [35].Khan I., Ndubuka N., Stewart K., Mckinney V., and Mendez I., “The use of technology to improve health care to Saskatchewan's First Nations communities,” Canada Communicable Dis. Rep., vol. 43, no. 6, pp. 120–124, 2017. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [36].Giuliani M. et al. , “User-centred design and evaluation of a tele-operated echocardiography robot,” Health Technol., vol. 10, no. 3, pp. 649–665, 2020. [Google Scholar]
- [37].Stańczyk B., Kurnicki A., and Arent K., “Logical architecture of medical telediagnostic robotic system,” in Proc. Int. Conf. Methods Models Automat. Robot., 2016, pp. 200–205. [Google Scholar]
- [38].Fuchtmann J. et al. , “COVID-19 and beyond: Development of a comprehensive telemedical diagnostic framework,” Intern. J. Comput. Assist. Radiol. Surg., vol. 16, no. 8, pp. 1403–1412, 2021. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [39].Jang S. M., Hong Y. J., Lee K., Kim S., Chiên B.V., and Kim J., “Assessment of user needs for telemedicine robots in a developing nation hospital setting,” Telemed. e-Health, vol. 27, no. 6, pp. 670–678, 2021. [DOI] [PubMed] [Google Scholar]
- [40].Wilhelm D. et al. , “Comparative evaluation of HD 2D/3D laparoscopic monitors and benchmarking to a theoretically ideal 3D pseudodisplay: Even well-experienced laparoscopists perform better with 3D,” Surg. Endoscopy, vol. 28, no. 8, pp. 2387–2397, 2014. [DOI] [PubMed] [Google Scholar]