Abstract
Venipuncture robots have superior perception and stability to humans and are expected to replace manual venipuncture. However, their use is greatly restricted because they cannot make decisions regarding the puncture sites. Thus, this study presents a multi-information fusion method for determining puncture sites for venipuncture robots to improve their autonomy in the case of limited resources. Here, numerous images have been gathered and processed to establish an image dataset of human forearms for training the U-Net with the soft attention mechanism (SAU-Net) for vein segmentation. Then, the veins are segmented from the images, feature information is extracted based on near-infrared vision, and a multiobjective optimization model for puncture site decision is provided by considering the depth, diameter, curvature, and length of the vein to determine the optimal puncture site. Experiments demonstrate that the method achieves a segmentation accuracy of 91.2% and a vein extraction rate of 86.7% while achieving the Pareto solution set (average time: 1.458 s) and optimal results for each vessel. Finally, a near-infrared camera is applied to the venipuncture robot to segment veins and determine puncture sites in real time, with the results transmitted back to the robot for an attitude adjustment. Consequently, this method can enhance the autonomy of venipuncture robots if implemented dramatically.
Keywords: puncture site decision, vein extraction, near-infrared vision, venipuncture robot
Footnotes
This work was supported by the National Natural Science Foundation of China (Grant No. U1813209) and Self-Planned Task of State Key Laboratory of Robotics and System (Harbin Institute of Technology) (Grant No. SKLRS202112B).
These authors contributed equally to this work.
References
- 1.Horattas M C, Trupiano J, Hopkins S, et al. Changing concepts in long-term central venous access: Catheter selection and cost savings. Am J Infect Control. 2001;29:32–40. doi: 10.1067/mic.2001.111536. [DOI] [PubMed] [Google Scholar]
- 2.Sampalis J S, Lavoie A, Williams J I, et al. Impact of on-site care, prehospital time, and level of in-hospital care on survival in severely injured patients. J Trauma-Injury Infect Crit Care. 1993;34:252–261. doi: 10.1097/00005373-199302000-00014. [DOI] [PubMed] [Google Scholar]
- 3.Armenteros-Yeguas V, Gárate-Echenique L, Tomás-López M A, et al. Prevalence of difficult venous access and associated risk factors in highly complex hospitalised patients. J Clin Nurs. 2017;26:4267–4275. doi: 10.1111/jocn.13750. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Lamperti M, Pittiruti M. Difficult peripheral veins: Turn on the lights. Br J Anaesthesia. 2013;110:888–891. doi: 10.1093/bja/aet078. [DOI] [PubMed] [Google Scholar]
- 5.Hulse E J, Thomas G O R. Vascular access on the 21st century military battlefield. J R Army Med Corps. 2010;156:S385–390. doi: 10.1136/jramc-156-04s-20. [DOI] [PubMed] [Google Scholar]
- 6.Chen A I, Balter M L, Maguire T J, et al. Deep learning robotic guidance for autonomous vascular access. Nat Mach Intell. 2020;2:104–115. doi: 10.1038/s42256-020-0148-7. [DOI] [Google Scholar]
- 7.Chen A I, Balter M L, Maguire T J, et al. Proceedings of 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) Hamburg, Germany: IEEE; 2015. Real-time needle steering in response to rolling vein deformation by a 9-DOF image-guided autonomous venipuncture robot; pp. 2633–2638. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Chen A, Nikitczuk K, Nikitczuk J, et al. Portable robot for autonomous venipuncture using 3D near infrared image guidance. Technology. 2013;1:72–87. doi: 10.1142/S2339547813500064. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Balter M L, Chen A I, Maguire T J, et al. Adaptive kinematic control of a robotic venipuncture device based on stereo vision, ultrasound, and force guidance. IEEE Trans Ind Electron. 2017;64:1626–1635. doi: 10.1109/TIE.2016.2557306. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Chen A I. Image-guided robotics for autonomous venipuncture. New Brunswick: Rutgers University; 2016. pp. 25–28. [Google Scholar]
- 11.Wang D H, Feng G L, Wang X, et al. Research on image segmentation algorithm based on features of venous gray value. Opto Electron Eng. 2018;45:180066. [Google Scholar]
- 12.Ji J, Zhao Y, Xie T, et al. et al. Automated vein segmentation from NIR images using a mixer-UNet model. In: Liu H, Yin Z, Liu L, et al.et al., editors. Intelligent Robotics and Applications. Cham: Springer International Publishing; 2022. pp. 64–75. [Google Scholar]
- 13.Lin G, Wang H, Sha M, et al. Proceedings of 2022 7th International Conference on Control and Robotics Engineering (ICCRE) Beijing, China: IEEE; 2022. Design of a Multi-data fusion intelligent venipuncture blood sampling robot; pp. 10–15. [Google Scholar]
- 14.Sha M, Wang H, Lin G, et al. Proceedings of 2022 2nd International Conference on Computer, Control and Robotics (ICCCR) Shanghai, China: IEEE; 2022. Design of multi-sensor vein data fusion blood sampling robot based on deep learning; pp. 46–51. [Google Scholar]
- 15.Zivanovic A, Davies B L. A robotic system for blood sampling. IEEE Trans Inform Technol Biomed. 2000;4:8–14. doi: 10.1109/4233.826854. [DOI] [PubMed] [Google Scholar]
- 16.Cheng Z, Davies B L, Caldwell D G, et al. A hand-held robotic device for peripheral intravenous catheterization. Proc Inst Mech Eng H. 2017;231:1165–1177. doi: 10.1177/0954411917737328. [DOI] [PubMed] [Google Scholar]
- 17.Kobayashi Y, Hamano R, Watanabe H, et al. Use of puncture force measurement to investigate the conditions of blood vessel needle insertion. Med Eng Phys. 2013;35:684–689. doi: 10.1016/j.medengphy.2012.12.003. [DOI] [PubMed] [Google Scholar]
- 18.Zhuang Y, Chen J, Liu Q, et al. Preliminary study on mechanical characteristics of maxillofacial soft and hard tissues for virtual surgery. Int J CARS. 2021;16:151–160. doi: 10.1007/s11548-020-02257-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Zhao Y, Ji J, Xie T, et al. et al. Vessel site selection for autonomous cannulation under NIR image guidance. In: Liu H, Yin Z, Liu L, et al.et al., editors. Intelligent Robotics and Applications. Cham: Springer International Publishing; 2022. pp. 88–99. [Google Scholar]
- 20.He T, Guo C, Jiang L, et al. Proceedings of 2021 IEEE International Conference on Real-Time Computing and Robotics (RCAR) Xining, China: IEEE; 2021. Automatic venous segmentation in venipuncture robot using deep learning; pp. 614–619. [Google Scholar]
- 21.Long J, Shelhamer E, Darrell T. Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Boston: IEEE; 2015. Fully convolutional networks for semantic segmentation; pp. 3431–3440. [DOI] [PubMed] [Google Scholar]
- 22.Ronneberger O, Fischer P, Brox T. U-Net: Convolutional networks for biomedical image segmentation. arXiv: 150504597
- 23.Valipour S, Siam M, Jagersand M, et al. Proceedings of 2017 IEEE Winter Conference on Applications of Computer Vision (WACV) Santa Rosa: IEEE; 2017. Recurrent fully convolutional networks for video segmentation; pp. 29–36. [Google Scholar]
- 24.Jain H, Deb K. An evolutionary many-objective optimization algorithm using reference-point based nondominated sorting approach, Part II: Handling constraints and extending to an adaptive approach. IEEE Trans Evol Computat. 2014;18:602–622. doi: 10.1109/TEVC.2013.2281534. [DOI] [Google Scholar]
- 25.Deb K, Jain H. An evolutionary many-objective optimization algorithm using reference-point-based nondominated sorting approach, Part I: Solving problems with box constraints. IEEE Trans Evol Computat. 2014;18:577–601. doi: 10.1109/TEVC.2013.2281535. [DOI] [Google Scholar]
- 26.Deb K, Pratap A, Agarwal S, et al. A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Trans Evol Computat. 2002;6:182–197. doi: 10.1109/4235.996017. [DOI] [Google Scholar]
- 27.Li W L, Xie H, Zhang G, et al. Hand—eye calibration in visually-guided robot grinding. IEEE Trans Cybern. 2016;46:2634–2642. doi: 10.1109/TCYB.2015.2483740. [DOI] [PubMed] [Google Scholar]
- 28.Wang G, Li W, Jiang C, et al. Simultaneous calibration of multi-coordinates for a dual-robot system by solving the AXB = YCZ problem. IEEE Trans Robot. 2021;37:1172–1185. doi: 10.1109/TRO.2020.3043688. [DOI] [Google Scholar]
- 29.Fu H, Xu Y, Wong D W K, et al. Proceedings of 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI) Prague: IEEE; 2016. Retinal vessel segmentation via deep learning network and fully-connected conditional random fields; pp. 698–701. [Google Scholar]
- 30.Al-Bander B, Williams B, Al-Nuaimy W, et al. Dense fully convolutional segmentation of the optic disc and cup in colour fundus for glaucoma diagnosis. Symmetry. 2018;10:87. doi: 10.3390/sym10040087. [DOI] [Google Scholar]
- 31.Balter M L, Chen A I, Fromholtz A, et al. Proceedings of 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) Daejeon, South Korea: IEEE; 2016. System design and development of a robotic device for automated venipuncture and diagnostic blood cell analysis; pp. 514–520. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Balter M L, Chen A I, Maguire T J, et al. The system design and evaluation of a 7-DOF image-guided venipuncture robot. IEEE Trans Robot. 2015;31:1044–1053. doi: 10.1109/TRO.2015.2452776. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Qiao Z, Li Y, Wu Z, et al. Automatic puncture system based on NIR image and ultrasonic image. In: Proceedings of International Conference on Mechanical, Aeronautical and Automotive Engineering (ICMAA). Malacca. 2017;108:15002. [Google Scholar]