Automatic force-controlled 3D photoacoustic system for human peripheral vascular imaging

Abstract

Photoacoustic (PA) imaging provides unique advantages in peripheral vascular imaging due to its high sensitivity to hemoglobin. Nevertheless, limitations associated with handheld or mechanical scanning by stepping motor techniques have precluded photoacoustic vascular imaging from advancing to clinical applications. As clinical applications require flexibility, affordability, and portability of imaging equipment, current photoacoustic imaging systems developed for clinical applications usually use dry coupling. However, it inevitably induces uncontrolled contact force between the probe and the skin. Through 2D and 3D experiments, this study proved that contact forces during the scanning could significantly affect the vascular shape, size, and contrast in PA images, due to the morphology and perfusion alterations of the peripheral blood vessels. However, there is no available PA system that can control forces accurately. This study presented an automatic force-controlled 3D PA imaging system based on a six-degree-of-freedom collaborative robot and a six-dimensional force sensor. It is the first PA system that achieves real-time automatic force monitoring and control. This paper's results, for the first time, demonstrated the ability of an automatic force-controlled system to acquire reliable 3D PA images of peripheral blood vessels. This study provides a powerful tool that will advance PA peripheral vascular imaging to clinical applications in the future.

1. Introduction

A variety of diseases exhibit changes in vascular morphology, perfusion, and oxygenation status in peripheral arteries and veins, such as peripheral vascular diseases (PVD), varicose veins, and superficial venous thrombosis (SVT) [1–4]. Peripheral vascular imaging is essential for characterizing these morphological and functional changes for disease diagnosis [5,6]. An ideal peripheral vascular imaging modality requires high contrast, high resolution, deep imaging depth, non-invasiveness, non-ionizing radiation, and real-time hemodynamic imaging ability. However, the current vascular imaging modalities suffer from various limitations. Digital subtraction angiography (DSA) is the gold standard for vascular imaging, but it involves ionizing radiation and exogenous contrast agent, which is unsuitable for patients with diabetes, impaired renal function, allergies, and pregnancy [7,8]. Other modalities like computed tomography angiography (CTA) [8,9] and magnetic resonance angiography (MRA) also require radiation and contrast agents [2]. Optical coherence tomography (OCT) is limited by low penetration depth (within 1 mm) [10]. And doppler ultrasound (DUS) is not sensitive to small blood vessels with low blood flow velocity [7]. Thus, these angiography modalities are not suitable for long-term imaging of chronic vascular diseases.

Photoacoustic (PA) imaging has become a promising medical imaging technology developing rapidly in recent years [11]. As a non-invasive and non-ionizing hybrid imaging modality, PA imaging combines the advantages of optical and ultrasound imaging and overcomes their limitations. It provides high optical contrast, high spatial resolution, and deep tissue penetration [12–16]. Due to the strong light absorption of hemoglobin, PA imaging offers high-contrast vascular imaging without using exogenous contrast agents [2,17]. Hence, it provides unique advantages in repetitive vascular imaging for chronic vascular diseases like PVD. In addition, based on the variation in the light absorption for different tissue chromophores, critical pathophysiological information, such as blood oxygenation, glucose concentration, metabolism, and other functional parameters, can be obtained using multispectral PA imaging [18,19]. Therefore, PA imaging is an ideal peripheral vascular imaging technique that can be game-changing for the diagnosis and treatment of various vascular diseases.

Due to clinical requirements for flexibility, affordability, and portability of PA imaging equipment, current PA imaging systems developed for clinical applications usually utilize dry coupling [20,21], however, it induces contact force between the probe and the skin [22]. The contact force applied to the blood vessels can lead to vascular morphology and perfusion alterations. This problem occurs in clinical ultrasound imaging, which also utilizes dry coupling. To investigate the effect of contact force, Gilbertson et al. developed a handheld ultrasound probe with force monitoring, which they used to perform ultrasound imaging of the brachial artery under three different forces [23]. In the ultrasound images, the brachial artery at 1N was nearly circular. However, with force increased to 5N, the artery almost completely collapsed, making it difficult to compare the images and thus affecting clinicians’ diagnosis. The diagnosis of vascular diseases depends on the reliability of the reconstructed vascular morphology, oxygenation, perfusion, and other parameters [6,24,25]. Alterations in PA vascular images can mislead the diagnosis and make it hard to compare repetitive imaging results of chronic vascular diseases. However, the effects of contact force on PA vascular imaging have never been investigated. The reason is that there is no available PA system with automatic force monitoring and control.

Currently, there are generally two types of PA systems for clinical peripheral vascular imaging: handheld and mechanical scanning [2,22]. In handheld systems, the acquisition status of the probe is manually controlled by the operator [26]. Hence, precisely controlling the position, direction, and contact force are infeasible. In addition, the long-term handheld operation can cause the operator potential musculoskeletal pain and strain [27]. Current mechanical scanning systems utilize stepping motors and fixed guide rails [28,29]. Thus, the motion of the transducer can only be performed in limited dimensions, which is inflexible to image peripheral blood vessels in different body parts to meet clinical needs. Moreover, it also lacks force sensing, which results in unknown and uncontrollable contact force between the probe and the skin, affecting the reliability of the results. Lacking the flexibility for different clinical applications, the precise control of imaging, and the ability to provide reliable results without interferences of contact force are the primary challenges for current PA systems in peripheral vascular imaging. These problems preclude the advancement of photoacoustic imaging to an accurate and reliable clinical vascular imaging technique. Therefore, developing a flexible PA imaging system with automatic force control is demanding for clinical peripheral vascular imaging.

In this paper, we proposed a novel automatic force-controlled 3D PA imaging system based on a six-degree-of-freedom (6-DOF) collaborative robot and a six-dimensional force sensor. This is the first PA system that achieves real-time automatic force monitoring and feedback control. It can provide 3D scanning on a non-flat surface with a selected contact force and online real-time tomographic PA image display during the scanning. Using this system, we demonstrated that the contact force could significantly affect the PA peripheral vascular imaging, and investigated the effect of the contact force on the size, shape, and contrast of the blood vessels in PA images. This system has the following advantages: (1) It avoids the alteration and unreliability of the PA vascular images caused by the unknown or uncontrolled contact force, which is an important advancement in the clinical transformation of PA peripheral vascular imaging; (2) This system uses a clinical ultrasound probe that makes it easier to be applied to clinical applications, and provides the potential to be combined with ultrasound for multimodal imaging; (3) This system utilizes a 6-DOF collaborative robot which can realize flexible and convenient setting of any scanning trajectories to image different parts of the body. And the robot offers a safety system allowing it to work safely beside operators, which makes it feasible to be used in clinical settings.

2. Materials and methods

2.1. System setup

Figure 1(A) and Fig. 1(B) show the system schematics and the photograph of the experiment, respectively. The laser source of the system is an optical parametric oscillator (OPO) laser (Beijing ZK Laser Co., Ltd., China) with a 100 Hz repetition rate and 660 to 1064 nm wavelength tunable range. The pulsed laser is delivered through an optical fiber bundle (CeramOptec, Germany). In this study, A 12 mm thick transparent solid-state ultrasound coupling pad (light transmission 81%) was placed between the US probe and the imaging target. This pad provided a gap that allowed light to shine on the region of interest below the probe to provide acoustic coupling without sacrificing light transmission. The optical fiber bundle was fixed beside the imaging target to create an illumination area much larger than the imaging region for a more uniform light distribution and to keep the distance between the light source and the imaging region nearly unchanged. It reduced the interference of light delivery change on the results of the quantitative investigation of the effect of contact forces on PA vascular images. We used a 128-element clinical linear array US probe (L12-5D, Acoustic Life Science Co., Ltd., China) with a center frequency of 7.5 MHz to detect the PA signals. The acquisition of PA data was performed by the Vantage256 system (Verasonics, USA). Our system realized real-time PA imaging at a 100 Hz frame rate, which was limited by the repetition rate of the laser. The PA pressure waves received by the linear array probe are too weak to provide sufficient image Signal-to-Noise Ratio (SNR) [30,31]. To address this problem, we used a 128-channel pre-amplifier (AMP128, Photosound, USA; 40 dB gain with -6 dB cut-off frequencies of 25 kHz and 35 MHz) between the probe and the Vantage256 system [32]. The digital delay pulse generator (Beijing ZK Laser Co., Ltd., China) sent triggers to the laser and the Vantage system simultaneously to precisely synchronize the laser output and PA data acquisition. It also sends trigger signals to control the movement of the collaborative robot (UR3, Universal Robot, Denmark). The robot can work safely with humans in the same environment [33]. At the robot's end, we installed a six-dimensional force sensor (HPS_FT_025, Hypersen Technologies Co., Ltd., China, maximum data output rate 2 kHz, precision 0.05N, Fz measuring range ±250N) through a flange for force monitoring and control. A 3D-printed holder was fixed on the force-measuring surface of the force sensor for clamping and fixing the US probe coaxially with the sensor. Different clamps can be installed for different US probes to meet the needs of various applications. This system's axial, lateral and tomographic resolution were quantified as 0.237 mm, 0.257 mm, and 0.274 mm, respectively (See Supplement 1 ^{(1.1MB, pdf)} ).

Fig. 1.

The composition of the system. (A) The system schematics. (B) The photograph of the experiment. PA: photoacoustic; US: ultrasound.

2.2. System workflow

The workflow of the system is shown in Fig. 2, which can be described as follows: (1) Set the center of the US probe end as the tool center point (TCP) of the robot and set the motion path of the TCP. Zero the value of the force sensor to eliminate the effects of gravity. (2) The robot arm moves to the first sampling position after receiving the trigger signal sent by the digital delay pulse generator. The force sensor monitors the contact force between the US probe and the imaging target and feeds back the force information to the robot in real time. In the force mode, UR3 can move along the selected direction with the specified contact force applied according to the feedback of the force sensor. (3) The digital delay pulse generator simultaneously sends triggers to the laser and the Vantage system to accurately synchronize the laser output and PA data acquisition. The pulsed laser illuminates the imaging target through the optic fiber bundle to generate the PA signals. (4) The PA signals are captured by the US probe, amplified by a 128-channel pre-amplifier, and acquired by the Vantage system. The robot simultaneously sends the force information and the coordinates of the TCP at this position to the Vantage host controller (a high-performance computer). (5) Reconstruction and display of the images are independent and concurrent with the acquisition [32]. The 2D PA images were reconstructed in MATLAB by the delay and sum reconstruction algorithm in the Vantage host controller [32] and presented in real-time at a 100 Hz frame rate. (6) The robot arm moves to the next sampling position after receiving the trigger signal sent by the digital delay pulse generator and repeats the above acquisition process. (7) After the US probe scans the whole imaging target according to the specified path under the robot's driving, we can obtain a series of 2D tomographic PA images, RF data, and corresponding force information and coordinates. (8) Use the Amira software to reconstruct a 3D PA image offline from these 2D tomographic PA images and their corresponding coordinates.

Fig. 2.

The system workflow. TCP: Tool center point; PA: Photoacoustic.

There are several advantages of applying this force sensor. Firstly, most robots offer limited force-sensing accuracy, far below the medical imaging requirements. For example, the force sensing accuracy of the UR3 robot in this system is 3.5N. By applying this force sensor, we improved the accuracy to 0.05N, which would realize the application of force-controlled medical imaging. Secondly, this force sensor provides the URCap, a plug-in that can easily and conveniently integrate the force sensor and the UR3 robot.

2.3. 3D force-controlled PA imaging accuracy evaluation

The force control can induce system error. To evaluate the accuracy of the 3D force-controlled imaging, we performed the phantom experiment. The transparent phantom was made of a 2% agarose solution. The upper surface of the phantom was inclined. Two approximately parallel pencil leads (0.7 mm diameter) were placed in the phantom (Fig. 4(A)), and the pencil lead used for evaluation was indicated with the red arrow. The US probe was placed vertically downward on the upper surface of the phantom. The medical ultrasonic gel was applied to the US probe and phantom for acoustic coupling. The US probe scanned along the pencil leads with automatic contact force control. We scanned 50 steps with a 0.5 mm interval. After scanning, we obtained fifty 2D PA tomographic images and their corresponding coordinates, which were used for 3D image reconstruction by Amira. For 3D reconstruction, target blood vessels in each 2D PA image were segmented with a set threshold, and the corresponding coordinates of the probe were applied to reconstruct the 3D PA image of the target blood vessels. In this experiment, the laser wavelength was 720 nm, and the pulse energy at the surface of the phantom was 7.8 mJ/cm², which was below the American National Standards Institute (ANSI) safety limit at this wavelength [34].

Fig. 4.

(A) The photograph of the phantom. (B) The 3D PA reconstruction of pencil leads. (C) Positioning accuracy: the distance from the center of the pencil lead (red arrow) in each 2D PA image to the fitted straight line.

The positioning accuracy of the force-controlled 3D PA imaging system was evaluated as follows: firstly, obtain the coordinate of the maximum in the 2D PA images, which is the center of the pencil lead. Secondly, use the least squares method to fit a straight line. Finally, calculate the distance from the center of the pencil lead in each 2D image to the fitted straight line, then the positioning accuracy can be obtained.

2.4. In vivo 2D PA imaging of peripheral blood vessels under different contact forces

To investigate the effect of contact force on PA imaging of peripheral blood vessels, we performed 2D PA imaging of the radial vein in the healthy volunteer's wrist under different contact forces. In standard clinical US imaging, the contact force between the ultrasonic transducer and the skin ranges from 5 to 20 N [35]. Therefore, the contact forces we inspected in this study were all within this range. By controlling the robot to change the contact forces, we perform 2D PA imaging of the same blood vessel in the volunteer's wrist under the contact forces of 6 N, 8 N, 10 N, 12 N, and 14 N, respectively. The system offers real-time PA tomographic imaging at a 100 Hz frame rate. To evaluate the contact-force-induced changes in PA blood vessel images, we acquired 100 frames of data at each contact force to reconstruct the averaged PA images, which reduces the changes in PA blood vessel images caused by the pulse circles. A normal resting heart rate for adults ranges from 60-100 bpm [36]. The subjects’ heart rates were measured before the experiments to ensure that data for at least one cardiac cycle could be acquired. The supplementary video Visualization 1 ^{(6MB, mp4)} shows the real-time dynamic change of the PA vascular images according to the change of the contract force as it changed from small to large and back again. The pulse-induced changes in PA blood vessel images can also be seen in this video. In this experiment, the laser wavelength was 720 nm, and the pulse energy at the skin surface was 6.4 mJ/cm², which was below the ANSI safety limit at this wavelength [34]. The volunteer and operators wore laser safety goggles to prevent eye damage from laser exposure.

2.5. Effect of contact force on the in-vivo human 3D PA peripheral vascular imaging

To quantitatively investigate the effect of contact force on the 3D PA peripheral vascular imaging, blood vessels in two parts of the body were imaged. We performed 3D PA imaging of the volunteer's median antebrachial vein in the wrist and the dorsal metacarpal veins in the hand back with automatic force control. During the scanning, the US probe was kept vertically downward, and the range of the contact force change was set to be within 1 N through the automatic force control of our system. To compare with the current 3D PA imaging system using a clinical US probe and mechanical scanning, we perform a linear PA scanning of the same blood vessel at the same position without force control. The US probe was kept vertically downward during the scanning. The step size for both scans was 0.1 mm. We recorded the force in the Z direction measured by the six-dimensional force sensor and the spatial coordinates of the probe in the Z direction during both scans. The robot moved the probe back to the start position after each scan to ensure both the scans with and without force control for the same subject began at the same position. The acquisition of PA signals, probe coordinates, and the contact forces at each sampling position, as well as the reconstruction of 2D tomographic PA images, were completed in real-time. To reduce the pulse-induced changes in the PA blood vessel images, we acquired 100 frames of data at each sampling position to reconstruct the averaged 2D PA images for accurate analysis of the contact-force-induced changes in blood vessels. The 2D PA images were normalized, and the target blood vessel was segmented. Then, Amira software was used to reconstruct the 3D PA image from these preprocessed 2D PA images and their corresponding coordinates. Besides, we used the maximum amplitude projection (MAP) algorithm [37] to obtain the top view of the 3D PA image. Both volunteers and operators wore laser safety goggles during the experiments to prevent eye damage from laser exposure. This study utilized the laser wavelength of 720 nm and the pulse energy density of 5.7 mJ/cm² at the skin surface, which was below the ANSI safety limit [34]. All in vivo experiments reported in this work were approved by Chongqing Medical University. All experiments were performed in accordance with relevant guidelines and regulations. All volunteers have signed the informed consent before performing the examinations.

2.6. In-vivo 3D PA imaging of the peripheral blood vessels in the human arm

To demonstrate the system's ability of 3D vascular imaging, we performed in-vivo PA imaging of peripheral blood vessels in the human arm with automatic force control and compared it to conventional linear scanning without force control. The experimental setup is shown in Fig. 3. The fiber bundle was attached to the probe. Since the position of the light source changed while scanning, the light fluence on blood vessels also changed. Therefore, the adaptive threshold was used in each 2D tomographic image to segment the blood vessel for 3D image reconstruction using Amira. The threshold was set as the half maximum pixel intensity in each 2D tomographic image. Each 2D image was normalized independently before the maximum amplitude projection of the 3D image using MATLAB. A skin removal algorithm was applied before segmentation and maximum amplitude projection [38].

Fig. 3.

Experimental setup for 3D PA imaging of peripheral blood vessels in the human arm.

3. Result

3.1. 3D force-controlled PA imaging accuracy evaluation

Figure 4(B) shows that the system proposed in this paper can precisely reconstruct the 3D PA image of the pencil lead phantom shown in Fig. 4(A). The pencil lead indicated by the red arrow was used for evaluation. The positioning accuracy was estimated using the abovementioned method, and the results are shown in Fig. 4(C). The maximum distance from the “pencil lead center” in each 2D PA image to the fitting line is 0.078 mm. Thus, the positioning accuracy is ±0.078 mm, much higher than the spatial resolution of photoacoustic tomography using a clinical US probe [18].

3.2. In vivo 2D PA imaging of peripheral blood vessels under different contact forces

Figure 5 shows the 2D PA images of the radial vein in the volunteer's wrists under different contact forces. All PA images were normalized. When the contact force was 6 N, the radial vein (indicated by the red rectangle) was nearly circular in the PA image. However, with the increase of the contact force, it gradually collapsed in the PA image. The radial vein was distorted from circular to oval as the contact force increased to 10 N, and when the contact force increased to 14 N, the vessel almost completely collapsed. Figure 5 demonstrates that the contact force can affect the morphology and contrast of the PA vascular images.

Fig. 5.

In vivo 2D PA imaging results of the human radial vein under different contact forces. (A), (B), (C), (D), and (E) are under 6N, 8N, 10N, 12N, and 14 N, respectively.

We calculated the longitudinal width (D1) of the radial vein (indicated by the red rectangle) along the black dotted line in the PA images under different contact forces. The threshold was set to be 20% maximum pixel value of all 2D PA images due to the low pixel intensities as the contact forces went larger than 10N. Figure 6(A) shows that the longitudinal width of the radial vein in the PA image gradually decreased with the increase of the contact force. When the contact force was 6N, the longitudinal width of the blood vessel was 1 mm. As the contact force increased to 14 N, the longitudinal width of the blood vessel became 0.3 mm, which decreased by 70%. In other words, the radial vein gradually collapsed as the contact force increased. The transverse width (D2) of the same radial vein in the PA images under different contact forces along the white dotted line was also calculated with the same method. As shown in Fig. 6(B), under the contact force of 6N, the transverse width of the blood vessel was 1.1 mm. When the contact force was rising to 8N, there was a rapid increase in the transverse width. Then continued increasing the force to 12N, the transverse width of the blood vessel gradually increased to 1.9 mm. However, as the contact force increased from 12N to 14N, the transverse width of the blood vessel decreased to 1.5 mm. The dynamic change of the area of the blood vessel region under different contact forces was also calculated with the same threshold of 20% maximum pixel intensity. Figure 6(C) presents the results of the calculated area of the same radial vein. When the contact force raised from 6N to 8N, the area of the blood vessel region gradually increased from 0.9 mm² to 0.98 mm². Continued to increase the force to 10N, the area of the blood vessel region decreased to 0.75 mm². As the force went up to 12N, the area of the blood vessel region raised up to 0.85 mm². However, when the force raised from 12N to 14N, the area of the blood vessel region rapidly decreased to 0.3 mm². In other words, the change of the blood vessel due to the rise of the contract force is not monotonical, and the blood vessel did not go through a linear change as the contract force gradually increased.

Fig. 6.

(A) and (B) are the longitudinal width along the black dotted line and the transverse width along the white dotted line in Fig. 5 under different contact forces; (C) The calculated area of blood vessel region according to different contact forces; (D) The SNR of each PA image under different contact forces. D1: The longitudinal width of the blood vessel. D2: The transverse width of the blood vessel. S: Calculated area of the blood vessel region. SNR: Signal-to-Noise Ratio.

In addition, we also calculated the Signal-to-Noise Ratio (SNR) of each PA image. SNR can be expressed in the following equation [39]:

$$\text{SNR = }\frac{\text{mean(object)}}{\text{std(background)}}$$ (1)

Where ‘std’ is the standard deviation, ‘mean’ is the mean value, ‘object’ refers to the pixels inside the red rectangular in Fig. 5 with a value greater than 20% of the overall maximum pixel intensity, and ‘background’ is the pixels of the background near ‘object’. Figure 6(D) presents the dynamic change of the SNR with the contact force. When the contact force was 6 N, the SNR was 35.65 dB. As the contact force increased to 14 N, the SNR decreased to 19.01 dB, reduced by 16.64 dB. There was a monotonical decrease in SNR as the contact force increased.

3.3. Effect of contact force on the in-vivo human 3D PA peripheral vascular imaging

Figures 7(A-F) are results for the median antebrachial vein in the wrist, and Fig. 7(G-L) present results for the dorsal metacarpal vein in the hand back. PA images of the wrist blood vessel were segmented with a threshold of 50% maximum pixel intensity for the 3D reconstruction shown in Fig. 7(A) and 7(B). Due to the low pixel intensity of the PA images for hand back imaging without force control, we cannot see the blood vessel with a 50% maximum threshold, therefore, 20% maximum pixel intensity was set as the threshold to segment the blood vessel for 3D reconstructions shown in Fig. 7(G) and 7(H). Figures 7(A) and 7(G) show the 3D PA reconstruction of the blood vessels acquired by automatic force-controlled scanning. The contact forces were set to be 5.5-6.5 N and 5-6 N (within a range of 1 N), which were applied on the non-flat surfaces of the wrist and hand back, respectively. The blood vessels were nearly cylindrical in the 3D PA images. The results show that the morphology of the blood vessels can be well preserved in force-controlled scanning. Figures 7(B) and 7(H) are the 3D PA reconstructions of the same blood vessels obtained by the linear scan without force control. In these images, the blood vessels almost collapsed, indicating a significant change in blood vessel morphology as the contact force was not controlled. Figures 7(C) and 7(I) are representative 2D tomographic PA images of the 3D PA imaging with the force-controlled scan; Fig. 7(D) and 7(J) are the results of the force-uncontrolled linear scan. The red arrows refer to the targeted blood vessels in the 3D reconstructions. Compared with the force-controlled situation in Fig. 7(C) and 7(I), the targeted blood vessels in Fig. 7(D) and 7(J) collapsed seriously, and the tissue was compressed due to uncontrolled contact force. Figures 7(E) and 7(K) present the top view MAP images of the 3D PA imaging with force control, and Fig. 7(F) and 7(L) are without force control. Compared with Fig. 7(E) and 7(K), the morphology and contrast of the blood vessel in Fig. 7(F) and 7(L) were significantly altered.

Fig. 7.

3D PA imaging of the median antebrachial vein in the wrist (A)-(F) and the dorsal metacarpal vein in hand back (G)-(L). (A) (G) and (B) (H) are the 3D PA reconstruction with and without force control, respectively; (C) (I) and (D) (J) are the representative tomographic images (at red dashed lines in E, K, F, L) of the 3D PA imaging with and without force control, respectively; (E) (K) and (F) (L) are the top-view MAP images (between the red dashed lines in C, I, D, J) of the 3D PA imaging with and without force control, respectively.

To further validate the system's ability of automatic force-controlled 3D PA imaging to produce reliable PA vascular images, the SNR and area of blood vessel region were calculated with the same threshold used for the 3D reconstructions above. We recorded the contact force and the probe's spatial coordinate in the Z direction during the scanning with force control (See Fig. 8(A) and 8(E)) and without force control (See Fig. 8(B) and 8(F)). Figures 8(A) and 8(E) show that the system can adaptively change the probe's spatial coordinate to control the contact force within a given range (<1 N) during the scanning. However, the traditional linear mechanical scanning of the non-flat surface results in uncontrollable changes in the contact forces. Figures 8(B) and 8(F) show that the contact forces increased and decreased, respectively, in the two experiments due to the non-flat contact between the probe and the surfaces of the two parts of the body. We calculated the cross-sectional area of the blood vessel regions where pixel intensity was larger than the thresholds with the two scanning modes. Figures 8(C) and 8(G) show that the area of the blood vessel regions in the PA images of the wrist and hand back were nearly constant with slight variations. The variation was respectively 0.07 mm² and 0.18 mm² for scanning with and without force control in the wrist. In hand back, the variation was 0.1 mm² and 0.39 mm² with and without force control, respectively. The variations in the area of blood vessel regions using the scanning without force control were much larger than using automatic force-controlled scanning. In the scanning without force control on the wrist, the area decreased (See the blue dots in Fig. 8(C)) as the contact force increased (See the red dots in Fig. 8(B)). Similarly, when scanning without force control on the hand back, the area increased (See the blue dots in Fig. 8(G)) as the contact force decreased (See the red dots in Fig. 8(F)). We also calculated the SNR of every 2D PA image in the two scanning modes according to Eq. (1). Figures 8(D) and 8(H) show that the SNR calculated with the blood vessel region can also be greatly affected by the contact force. When scanning with force control, the average SNR was 47.81 dB and 49.34 dB for the wrist and the hand back, respectively. When linear scanning without force control, the average SNR was 27.27 dB and 27.49 dB.

Fig. 8.

Changes of the area and SNR of PA blood vessel images during the force-control and force-uncontrolled scanning of the median antebrachial vein in the wrist (A)-(D) and the dorsal metacarpal vein in hand back (E)-(H). (A) (E) and (B) (F) are the change of the contact force and the spatial coordinate of the probe in the Z direction during the scanning with and without force control, respectively; (C) (G)The cross-sectional areas of the blood vessel region where pixel values larger than the thresholds with the two scanning modes; (D) (H) The SNR calculated with the blood vessel region where pixel values larger than the thresholds.

3.4. In-vivo 3D PA imaging of the peripheral blood vessels in the human arm

The peripheral blood vessel in the human arm was imaged using our automatic force-controlled PA system and compared to the conventional 3D linear scanning without force control. Blood vessels were segmented with the adaptive threshold in each 2D tomographic image for 3D reconstruction. Figures 9(A) and 9(B) present the 3D PA reconstruction of the blood vessels (from two angles of view) acquired with and without force control, respectively. Figures 9(C) and 9(D) are the top-view MAP images of the blood vessels obtained with and without force control, respectively. Skin signals were removed before the normalization of each 2D tomographic image to form the MAP images, which eliminated the interference of the skin and enhanced the blood vessel. Figure 9(G) shows the representative original 2D tomographic images at the white dashed lines in Fig. 9(C) and 9(D). In Fig. 9(G), the skin was not removed, and the blood vessels were not enhanced. The contact forces during the force-controlled and force-uncontrolled scans are shown in Fig. 9(E) and 9(F), respectively. Since the surface of the arm is non-flat, the contact force increased during the linear scanning without automatic force control. The results show that automatic force-controlled PA imaging can offer better peripheral vascular images than conventional force-uncontrolled scanning. The morphology of the blood vessels can be better preserved with automatic contact force control. Whereas, in the conventional force-uncontrolled scanning, the 2D tomographic images of the blood vessels were expanded in the transverse direction and compressed in the longitudinal direction as the blood vessels were squashed when the contact force increased. In addition, the vascular structure was also distorted due to uncontrolled contact forces.

Fig. 9.

In-vivo 3D PA imaging of the peripheral blood vessels in the human arm with and without automatic force control. (A) and (B) are 3D blood vessel images (from two angles of view); (C) and (D) are top-view MAP images; (E) and (F) are the change of the contact forces during the scanning; (G) is the 2D tomographic images at the white dashed lines in (C) and (D), the red arrows indicate the blood vessels and the yellow arrow indicates an air bubble in the PA images.

4. Discussion

In clinical PA imaging of human peripheral blood vessels, dry coupling provides good practice for the requirement of flexibility, affordability, and portability of the PA imaging equipment. Although this coupling method is widely used, it inevitably induces unknown and uncontrolled contact forces between the probe and the skin, causing alterations in vascular morphology and perfusion. Such effect would result in altered shape, size, and contrast of PA blood vessel images, leading to inaccurate assessment/imaging of functional parameters such as oxygen saturation, metabolism, and perfusion [24,25]. Alterations in PA vascular images would raise complications in the diagnosis and the comparison of repetitive imaging results for chronic vascular diseases. However, the effect of contact forces on PA imaging remains unclear. In this study, we verified the effect of contact force on PA vascular morphology and contrast by imaging the peripheral blood vessels in the wrist under different contact forces (see Fig. 5). With the increase of the contact force, the blood vessels gradually collapsed. Specifically, when the contact force increased from 6 to 14 N, the results of Fig. 6(A) and 6(C) show that the longitudinal width and the SNR of the blood vessel were reduced by 70% and 16.64 dB, respectively. And the results of Fig. 6(B) show that the transverse width of the PA blood vessel image first increased as the shape of the blood vessel was distorted from circular to oval. However, continuing to increase the contact force, the SNR dropped significantly due to the reduced blood vessel size and the hemoglobin, leading to a decrease in the transverse width of the PA blood vessel image. This study demonstrated that contact force could significantly affect PA peripheral vascular imaging.

To investigate the effect of the unknown and uncontrolled contact force on the current mechanical-scanning 3D PA systems using the clinical US probe and dry coupling. We performed a traditional linear scan of the peripheral blood vessels in the wrist and hand back without force control. Figure 7 and Fig. 8 demonstrated that the contact force could significantly affect the vascular morphology and contrast in 3D PA imaging. Thus, the results are not predictable, reliable, or repeatable without knowing and controlling the contact force. This problem has become the primary challenge that precludes the advancement of PA peripheral vascular imaging to clinical applications.

To address this challenge, we proposed an automatic force-controlled 3D PA imaging system, which is the first PA system to achieve real-time automatic force monitoring and control. The results in Fig. 7, Fig. 8, and Fig. 9 demonstrated that our system could significantly improve the 3D PA peripheral vascular imaging compared to the current force-uncontrolled mechanical scanning. With either a fixed light source or a light source attached to the scanning probe, our automatic contact-force-controlled system can provide better PA peripheral vascular images than imaging without force control. With real-time automatic monitoring and control of contact forces, operators can easily set a contact force to obtain reliable PA vascular images, and use that contact force to perform repetitive imaging for more accurate quantification of chronic vascular diseases. Thus, this automatic force-controlled 3D PA system has made an important advancement in the clinical transformation of PA peripheral vascular imaging. In addition, this new system was developed on a 6-DOF cooperative robot, making it more portable and flexible. The current mechanical scanning systems using the stepping motor and the fixed moving rail can only scan in limited dimensions. The 6-DOF cooperative robot in this new system offers a drag teaching function allowing the operators to set any scanning trajectories easily and flexibly for imaging different parts of the body. It overcomes the scan-postures limitation of traditional mechanical scanning systems. Thus, this system provides a new path for more convenient and flexible 3D PA imaging, by combining the humans’ intelligent advantages in path planning and robotic arms’ advantages in force sensing control, scanning accuracy, and scanning efficiency. Furthermore, the application scenarios of PA imaging usually require human-machine cooperation, and safety during scanning is paramount. Compared with traditional mechanical scanning, cooperative robots can perceive the surrounding environment and change their behavior according to the changes in the surrounding environment, such as force. The mechanical arm's function of force control and collision detection allows it to work safely beside operators, making it feasible to be used in clinical settings. On the other hand, although handheld systems can offer more flexibility, they cannot realize automatic force control. The operator manually controls the probe, making it impossible to keep a constant contact force during scanning. The manually controlled operation can also cause potential musculoskeletal pain and strain on the clinicians [27]. This new system can compensate for the lack of automatic control of handheld systems. Moreover, this system combines six-dimensional high accuracy (0.05N) force sensor with the six-degree-of-freedom collaborative robot, which provides the potential to be further developed as automatic adaptive 3D imaging for different applications. Also, it is easier to be applied in clinical applications and provides the potential to be combined with ultrasound for multimodal imaging, as it utilizes the clinical ultrasound probe.

Although the current development of this new system realized real-time automatic force monitoring and control to provide reliable 3D PA peripheral vascular imaging, it still has some limitations, which we hope to optimize in future work. Firstly, the current system only utilized force sensing in the vertical direction. In this future work, we will further develop the system to achieve automatic six-dimensional force-controlled adaptive scanning. Secondly, the pre-amplifier in the current system is not compatible with US imaging. To realize the multimodal imaging of the system, we are developing a multi-channel pre-amplifier with high-voltage switches that will allow PA-US imaging. Thirdly, optimizing the light delivery will further improve the system's performance. At last, this study only investigated the effect of contact force on PA vascular images. To further establish the potentials of the automatic force control, other properties, such as optical attenuation, oxygen saturation, and blood perfusion, will also be investigated under different contact forces using the automatic force-controlled system.

5. Conclusion

In summary, we have successfully developed an automatic force-controlled 3D photoacoustic imaging system based on a 6-DOF collaborative robot and a six-dimensional force sensor. It is the first PA system that achieves real-time automatic contact force monitoring and control. Using this system, we validated that contact forces can significantly affect the PA vascular morphology and contrast by experiments under different contact forces and force-uncontrolled 3D scanning. This study proved that automatic force control is necessary for PA vascular imaging systems which utilize dry coupling. For the first time, we demonstrated that the automatic force-controlled system could provide 3D PA vascular imaging with better-preserved morphology and contrast than conventional scanning without force control. We expect that with the automatic force control, the PA 3D imaging using the clinical ultrasound probe and dry coupling will be more reliable and repeatable. This study provides a powerful tool that will advance PA peripheral vascular imaging to clinical applications in the future.

Funding

National Natural Science Foundation of China10.13039/501100001809 (62101083, 62201103); Natural Science Foundation of Chongqing10.13039/501100005230 (cstc2021jcyj-msxmX0739, cstc2021jcyj-msxmX0104); China Postdoctoral Science Foundation10.13039/501100002858 (2020M683260).

Disclosures

The authors declare no conflicts of interest.

Data Availability

Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.

Supplemental document

See Supplement 1 ^{(1.1MB, pdf)} for supporting content.