Development and preliminary testing of a prior knowledge-based visual navigation system for cardiac ultrasound scanning

Abstract

Purpose

Ultrasound is widely used to diagnose disease and guide surgery because it is versatile, inexpensive and radiation-free. However, image acquisition is dependent on the operation of a professional sonographer, which is a difficult skill to learn for a wider range of non-sonographers.

Methods

We propose a prior knowledge-based visual navigation method to obtain three important standard ultrasound views of the heart, based on the sonographer’s skill learning and augmented reality prompts. The key information about the probe movement was captured using vision-based tracking and normalisation methods on 14 volunteers, based on a professional sonographer’s practice. An augmented reality-based navigation method was then proposed to guide operators with no ultrasound experience to find standard views of the heart in a second set of three volunteers.

Results

Through quantitative analysis and qualitative scoring, the results showed that the proposed method can effectively guide non-sonographers to obtain standard views with diagnostic value.

Conclusion

It is believed that the method proposed in this paper has clear application value in primary care, and expansion of the data will allow the accuracy of the navigation to be further improved.

Introduction

Ultrasound is widely used in clinical analysis, using high-frequency sound waves to image human organs in real time for disease diagnosis. Compared to CT and MRI, ultrasound is inexpensive, safe, radiation-free and provides real-time imaging. However, the main barrier to reliable imaging is lack of training. The sonographer must first determine the approximate area of interest to be scanned and then perform a series of operations, including moving the ultrasound probe within the area of interest, placing the probe at a specific angle, applying an appropriate force to ensure safety and image clarity, and adjusting the probe while observing the ultrasound image in real time, to obtain the desired ultrasound image [1]. Therefore, ultrasound scanning is a difficult skill to learn and the number of sonographers is small, especially in some of the remote areas of developing countries [2, 3]. In recent years, teleconsultation and remote ultrasound robotics have attracted attention. Although the introduction of these technologies expands the possibilities for scans that were previously only available on-site, the accuracy of the diagnosis still depends on the experience of the sonographer [4, 5]. In addition, teleconsultation still requires the real-time involvement of specialists, which is impractical in many cases because specialists are usually busy and it may even be difficult for them to perform regular routine diagnosis, let alone teleconsultation.

With the rapid development of portable ultrasound equipment in recent years, it is natural to consider how ultrasound can be made available to primary care physicians. These primary care physicians can be general practitioners, community health workers, etc. Meanwhile, AI technology has developed rapidly over the years, and there has been a large amount of research on algorithms related to the automatic analysis and interpretation of ultrasound images, as reviewed in [6–8]. This also means that if the relevant images can be acquired, the subsequent interpretation is likely to be automated. To date, the greatest challenge is how to get less experienced sonographers or even non-sonographers to place the probe in the correct location and adjust it to the correct posture. Further analysis of the ultrasound acquisition procedure shows that applying the appropriate force is relatively less difficult than finding the correct probe position. This is because the amount of force applied directly affects the contrast of the image, which can be mastered by the operator through effective training. However, finding the correct probe position, especially in relation to the body, is very difficult and requires many years of practice.

Based on the above background and practical challenges, we believe that there could be a navigation system during ultrasound acquisition that could guide the operator in the placement of the probe. The work based on this idea is overall very limited and here we present some typical representatives. In an attempt to develop automatic navigation systems, Mustafa et al. proposed a method to estimate the epigastric region by using the position of the umbilicus and the nipple in the image, and then utilized a robot to automatically scan the liver based on the designed ultrasound scanning protocol [9]. This navigation method can only observe part of the anatomical structure of the liver and the scanning protocol is blind, which cannot ensure that the desired anatomy will be observed. In addition, the position of the navel may be difficult to identify in some cases. Bimbraw et al. proposed an augmented-reality-based approach to guide the position of lung ultrasound scanning [10]. An ArUco marker was mounted on the top of the ultrasound probe to obtain the probe tip’s position and guide the scan. Further, Ma et al. proposed a robotic lung ultrasound system that includes automatic acquisition of the position and execution of the probe based on the work of Bimbraw et al. [11]. An RGB-D camera was used to recognize the scanning position of the patient utilizing a learning-based human pose estimation method and a seven-degree-of-freedom (DoF) robotic arm was employed to position the probe at the specified position. But these two methods have not yet been tested on humans to verify their effectiveness. In addition, lung ultrasound acquisition is relatively straightforward by partitioning the thorax with relatively little variation in probe posture. These studies have not yet given an indication of how the method can be extended to other organs.

Although the above work is instructive, further implementation of cardiac ultrasound navigation is more challenging. This is primarily due to the complex anatomical structure of the heart, which requires not only probe positioning but also accurate manipulation of the probe angle during imaging [12, 13]. In addition, cardiac ultrasound is rarely incorporated into routine physical examinations and is not mandatory for physicians, suggesting limited capacity to perform such scans in primary care settings. Therefore, the focus of this paper is set on how to propose a navigation method to assist the operator in acquiring standard ultrasound views of the heart. Three important standard ultrasound views of the heart for diagnosis are selected as the study target, including the Apical Four Chamber (A4C) view, Parasternal Long Axis (PLAX) view and Parasternal Short Axis (PSSA) view [14]. A straightforward idea for the conceived navigation system is that it should contain two core modules, one of which is the collection of the experts acquisition experience, i.e., where and in what position relative to the body the probe should be placed for patients of different body sizes. The second is augmented reality guidance, i.e., the ability to implement human–machine interaction in some visual way to guide the operator to position the probe.

In this study, we propose a prior knowledge-based visual navigation method to obtain the standard ultrasound views of the heart to assist manual ultrasound scanning. The implementation of the navigation is based on the learning of physician skills. To gain this knowledge, data representing the ultrasound probe’s coordinates relative to the human body and the angle of the probe were collected based on the demonstration of a professional sonographer. Furthermore, an augmented reality (AR)-based navigation approach is proposed to provide a visual-based user interaction to assist the operator in positioning a probe. The effectiveness of the overall workflow was validated by a volunteer study. To the best of our knowledge, the present study is the first to address the relevant issues and perform in-vivo testing.

Materials and methods

Motion tracking of the ultrasound probe

The aim of this section is to propose a simple and easy-to-use acquisition method for the sonographer to record the probe motion information. Although there are many well-established methods for motion tracking, such as using optical tracking systems or electromagnetic tracking systems, such devices are usually complex and not conducive to deployment in hospitals and everyday environments. In recent years, vision-based pose tracking methods have become increasingly sophisticated, which greatly simplifies the deployment and operation of the devices. Therefore, the simplest monocular camera and marker-based tracking approach is adopted in this paper to facilitate the deployment, use, and interface of the subsequent camera-based navigation methods. It should be noted that the accuracy of the machine vision-based pose tracking method used in this study is not our main concern. This has been analyzed in many vision-related studies [15, 16], and we are more interested in its specific application to the process of operational acquisition by ultrasound scanning physicians.

To obtain the position and orientation of the ultrasound probe when collecting the standard ultrasound views of the heart, a ChArUco cube [17, 18] was installed on top of the ultrasound probe with a mono camera (RMONCAM G180, 1920*1080, Shenzhen, China) set up above the human body for probe motion tracking, as shown in Fig. 1a. The camera was calibrated with a checkerboard after collecting 38 images from different perspectives and using the standard method provided by OpenCV. The ChArUco marker has 48 corners with a size of 5 × 5, which is composed of ArUco markers with 4 × 4 marker dictionary. Compared to ArUco, the use of the ChArUco cube can improve the detection stability and accuracy, as well as ensure the success of detection under partial occlusion. Additionally, since the ArUco marker is a two-dimensional code, it is difficult for camera to detect the corners when the tilt angle of the marker is large. When multiple ChArUco markers are detected simultaneously, the marker with a smaller tilt angle is selected for pose estimation. The speed of ChArUco marker detection can reach 10 Hz, which is sufficient to meet the requirements of real-time detection.

Fig. 1.

The scenario of the data collection: a schematic diagram of probe tracking and the frame transformation: (1) $T_{\mathit{marker}}^{\mathit{cam}}$ and (2) $T_{\mathit{tip}}^{\mathit{marker}}$, b tracking of the initial (upper) and final (bottom) probe locations and c the results of the DensePose algorithm with the U, V coordinates and the contour plot

Skill learning from sonographer’s demonstration

This section aims to present the detailed methods for the collection of the sonographer’s probe positioning-related operational skills. This is to include both the absolute poses of the probe obtained based on the method presented in the previous section, and the relative relationship between the probe position and the body coordinates, taking into account the differences in human body size. As shown in Fig. 1b, the human body is in the supine position, and the probe mark is used to indicate the imaging direction. The red, green and blue line represent the X-, Y- and Z-axis, respectively. The initial location of the probe $T_{\mathit{initial}}^{\mathit{cam}}$ in the camera frame is the state when the probe’s X-, Y- and Z-axis are estimated to be perpendicular to the planes L, A, and I, respectively. The final pose of the probe in the camera frame, when an ideal ultrasound view is obtained (such as the A4C view shown in Fig. 1(b) as examples), is denoted as $T_{\mathit{final}}^{\mathit{cam}}$. Both $T_{\mathit{initial}}^{\mathit{cam}}$ and $T_{\mathit{final}}^{\mathit{cam}}$ can be obtained by:

$$T=T_{\mathit{marker}}^{\mathit{cam}}·T_{\mathit{tip}}^{\mathit{marker}}$$ 1

where $T_{\mathit{marker}}^{\mathit{cam}}$ is a homogeneous matrix representing the transformation from the marker to the camera frame, which can be obtained using the method introduced in the previous section. $T_{\mathit{tip}}^{\mathit{marker}}$ denotes the transformation from the probe tip to the marker frame. The ChArUco cube is made by 3D printing based on the 3D probe model obtained by scanning, so $T_{\mathit{tip}}^{\mathit{marker}}$ can be obtained from the computer-aided design software. The corresponding coordinate frames can be found in Fig. 1a. The marker frame is set in the same direction as the probe tip frame but with a different origin. The initial location of the probe $T_{\mathit{initial}}^{\mathit{cam}}$ acts as the reference frame and the final pose of the probe relative to the reference frame $T_{\mathit{final}}^{\mathit{initial}}$ can be obtained by:

$$T_{\mathit{final}}^{\mathit{initial}}=T_{\mathit{initial}}^{cam-1}·T_{\mathit{final}}^{\mathit{cam}}$$ 2

To obtain the probe’s positioning information relative to the human body when collecting the standard ultrasound views of the heart, the DensePose algorithm is used to map all human pixels of an RGB image to the 3D surface of the human body. DensePose is a deep learning algorithm based on the architecture of Mask-RCNN, which can perform per-pixel classification of each part of the human body and coordinates regression within the part [19–21]. In our case, the region of the chest was extracted, and the UV coordinates within the region were densely regressed to between 0 and 1, which is the normalized representation relative to the region as shown in Fig. 1c and the following related figures. Detecting the ChArUco marker is also used to obtain the pixel position of the probe tip, which can be expressed by:

$$\left[\begin{array}{c}ti{p}_{\mathit{uv}}\\ 1\end{array}\right]=cmtx·T_{\mathit{marker}}^{\mathit{cam}}·\left[\begin{array}{c}ti{p}_{\mathit{xyz}}\\ 1\end{array}\right]$$ 3

where $ti{p}_{\mathit{uv}}={\left[ti{p}_u,ti{p}_v\right]}^T$ denotes the pixel location of the probe tip in the UV coordinates. The cmtx is a 3-by-4 matrix obtained by the camera calibration, that contains the intrinsic parameters of the camera. $ti{p}_{\mathit{xyz}}=\left[ti{p}_x,ti{p}_y,ti{p}_z\right]$ denotes the 3D coordinates of the tip location in the marker frame, which are determined by the installation position of the ChArUco marker. Then the pixel location of the probe tip could be mapped to the UV coordinates by DensePose.

Augmented reality based navigation method

The purpose of this section is to use the abov-recorded physician’s operation information to design a visually based navigation method to guide manual ultrasound scanning for inexperienced operators. To achieve that, pixel-based location guidance and Euler angle-based orientation guidance methods are combined. As an example, the position navigation system for the A4C view is shown in Fig. 2a. The following is the expected workflow. The normalized probability density for acquiring the desired ultrasound view is generated and displayed as an overlaid heatmap with values between 0 and 1 (areas with value below 0.2 are ignored) to the operator on a computer screen using the DensePose algorithm and the statistical results obtained from the previous probe tracking methods. The probe is placed at the specified initial posture explained in the previous section and a random initial position by the operator, which is then recorded as the reference frame. The operator is expected to move the probe from its initial position to the red area on the overlaid heatmap, while its current pixel-wise location is indicated in green.

Fig. 2.

The visual form of the navigation system presented to the operator using a real case of the A4C view as an example: a the expected position and b the orientation of the probe

For the navigation of the probe’s orientation, the pose of the probe is decomposed into Euler angles in the order of XYZ. As indicated by an example of the orientation navigation system shown in Fig. 2b, the probe is treated as a vector with an initial state of (0, 0, z) and the operator is expected to tilt the probe until the green arrow reaches the center of the red cone at the bottom. The vector forms a sphere in cartesian space, and each point on the sphere corresponds to the rotation transformation in the order of XY. In the spherical space, the red cone represents the range of the vector where the desired ultrasound view may be collected, and the green arrow represents the position of the vector where the current probe is located. The operator is further expected to rotate a certain angle around the Z axis of the probe to make the green arrow reach the center of the red sector on the top disk. The range of the red cone and sector can be determined using the statistical results obtained from the operator’s skill collection and the real-time tracking of the ultrasound probe can be achieved using the vision-based approach, both of which have been explained in the previous sections.

Results

Experimental results of the probe positioning

The purpose of this section is to present the statistical data on the collected positions and orientations of the ultrasound probe during the cardiac scans, aiming to demonstrate the feasibility of our navigation method. In this study, 14 male volunteers aged 23 on average were recruited for data collection. The mean, standard deviation, minimum, and maximum value of BMI for the 14 male volunteers are 22.6, 3.2, 18.7 and 29.3 ${kg/m}^2$. A professional sonographer with more than ten years of experience was invited to perform the cardiac examination. We have obtained institutional review board (IRB) approval from Aerospace Central Hospital before conducting human subject experiments and the number is ‘Ethical Review No. (079) of Hangtian Center Hospital 2022’.

After removing the outliers, the statistics of the roll, yaw, and pitch angles and the UV coordinates are displayed in Tables 1 and 2. The boxplot of the Euler angles and UV coordinates are shown in Fig. 3. Min, Max, Mean, and Standard Deviation in Tables 1 and 2 denote the minimum, maximum value, mean, and standard deviation of the 14 data sets collected for probe positions and orientations, respectively. Taking the A4C view as an example, A4C-R, A4C-P, and A4C-Y represent the roll, pitch, and yaw angles of the probe when acquiring this particular view. Similarly, A4C-U and A4C-V represent the U and V coordinates of the probe relative to the human body when acquiring the A4C view. The UV coordinates were fitted by a two-dimensional normal distribution, and the correlation coefficients were 0.240, 0.321, and −0.160 for the three cardiac ultrasound views investigated in this study. The statistical results from Tables 1 and 2 and Fig. 3 show that the probe positions and orientations for the same ultrasound view may vary between different individuals, but they generally fall within a narrow range. This suggests that there is a high probability of obtaining the desired ultrasound view for different individuals within this range, which strongly supports the feasibility of our navigation method. The normalized probability density of the expected position was mapped onto the surface of the human body and illustrated in the form of a heatmap with values between 0 and 1, as shown in Fig. 4. Areas with a probability density lower than 0.2 times of the maximum were ignored during the process.

Table 1.

Statics of the probe’s roll, yaw, and pitch angles collected from the volunteer experiment

	Min (∘)	Max (∘)	Mean (∘)	Standard Deviation (∘)
A4C-R	− 44.2	− 29.0	− 36.2	4.8
PLAX-R	2.5	25.0	15.8	6.2
PSSA-R	0.1	17.5	8.4	5.2
A4C-P	18.6	37.4	27.8	5.7
PLAX-P	− 7.7	7.3	1.9	4.9
PSSA-P	− 10.3	2.6	− 3.6	4.3
A4C-Y	148.7	183.0	167.5	9.4
PLAX-Y	27.5	65.6	45.4	12.1
PSSA-Y	116.8	139.1	125.7	7.1

Table 2.

Statics of the probe’s UV coordinates collected from the volunteer experiment

	Min	Max	Mean	Standard deviation
A4C-U	0.289	0.356	0.326	0.017
PLAX-U	0.232	0.283	0.254	0.013
PSSA-U	0.249	0.310	0.277	0.016
A4C-V	0.289	0.382	0.328	0.029
PLAX-V	0.391	0.455	0.430	0.018
PSSA-V	0.379	0.460	0.420	0.026

Fig. 3.

Boxplot of the a Euler angels and b UV coordinates collected from the experiment

Fig. 4.

The heatmap of the expected position for a–c the A4C, PLAX, PSSA view and d all of the three views fused together

As shown in Fig. 4, the UV coordinates can be transformed into the range of translation along the X and Y axes in the probe tip frame through camera model, which can be achieved by:

$$\left[\begin{array}{c}edg{e}_{\mathit{uv}}\\ 1\end{array}\right]=cmtx·\left[\begin{array}{c}edg{e}_{\mathit{xyz}}\\ 1\end{array}\right]$$ 4

where $edg{e}_{\mathit{uv}}={\left[edg{e}_u,edg{e}_v\right]}^T$ denotes the pixel location of the edge point of the UV coordinate range, which can be easily obtained. $edg{e}_{\mathit{xyz}}={\left[edg{e}_x,edg{e}_y,edg{e}_z\right]}^T$ denotes the position of the edge point in the camera frame. $edg{e}_z$ is the same as the Z coordinate of the probe tip in the camera frame with the probe at the desired position, and the points in the target area can be approximately considered to be in the same plane. Therefore, $edg{e}_x$ and $edg{e}_y$ can be obtained by solving the Eq. 4. The translation range can be obtained by using the two edge points of the upper left corner and the lower right corner.

The roll, yaw, and pitch angles were fitted with normal distributions, respectively, and the ranges of Euler angles were determined by the $2\sigma$ criterion. The visualized Euler angle ranges of the three cardiac ultrasound views are shown in Fig. 5. The fan-shaped red area shows the range of the yaw angle, which indicates the probe’s rotation along its Z-axis. The cone-shaped red area represents the range of the pitch and roll angles, indicating where the Z-axis of the probe is pointing.

Fig. 5.

The visualized Euler angle ranges of the a A4C, b PLAX and c PSSA view

Experimental results of the navigation

The purpose of this section is to examine the effectiveness of the proposed navigation method from two perspectives: first, to determine whether the probe position and orientation corresponding to the desired ultrasound view collected during the validation experiments are within the previously established statistical range; second, to ensure that the desired ultrasound view obtained using the proposed navigation method has diagnostic value. Three new volunteers who had not attended the previous data collection experiment were recruited for the validation experiment. The BMI values of the three volunteers participating in the validation experiments are 21.1, 23.7 and 25.4 ${\text{kg/m}}^2$. The experiment was conducted according to the instructions of the proposed ultrasound navigation system by an operator who had no ultrasound experience.

Due to the relatively slow speed of DensePose, it may not be feasible to run DensePose in real time. However, it is acceptable to assume that the patient does not move significantly during the ultrasound examination and run the algorithm only once at the beginning of the experiment. As a result, the final frame rate of the guidance procedure can reach 10 Hz, which is influenced by the speed of marker detection. In this study, the navigation procedure is implemented using Python programming and executed on an Intel i7-11800H @2.30 GHz processor. Before the experiment, the operator was informed of the navigation system and received a quick training on the characteristics of the three acquired ultrasound views. It should be noted that this training is very basic and does not contain any specific instructions demonstrated on the recruited volunteers. During the experiment, the operator would place the probe according to the prompts of the navigation system and the basic knowledge that the operator has learned after training. When the operator thought he had obtained a reasonable image, the current Euler angles and UV coordinates of the probe were collected. The visualization of these new cases relative to the statistical range of the previous experience data is shown in Fig. 6 and Fig. 7. The detailed data are summarized in Tables 3 and 4.

Fig. 6.

The orientation of the ultrasound probe for the three new cases represented in Euler angles, in comparison with the previously collected ranges: the a A4C, b PLAX and c PSSA view

Fig. 7.

The position of the ultrasound probe for the three new cases represented in UV coordinates, in comparison with the previously analyzed distributions: a A4C, b PLAX and c PSSA view

Table 3.

The final recorded Euler angles of the probe for three volunteers

	View	R (∘)	P (∘)	Y (∘)
Case1	A4C	− 29.1	33.3	148.0
	PLAX	13.1	− 2.7	42.6
	PSSA	10.3	− 8.5	130.3
Case2	A4C	− 25.2	27.7	147.2
	PLAX	18.1	− 6.4	30.3
	PSSA	13.6	− 5.6	124.8
Case3	A4C	− 20.8	26.4	146.9
	PLAX	16.5	− 3.6	40.9
	PSSA	8.9	8.3	132.2

Table 4.

The final recorded UV coordinates of the probe for three volunteers

	View	U	V	Probability Density
Case1	A4C	0.302	0.333	0.378
	PLAX	0.247	0.410	0.619
	PSSA	0.251	0.394	0.173
Case2	A4C	0.314	0.337	0.735
	PLAX	0.258	0.460	0.197
	PSSA	0.254	0.454	0.210
Case3	A4C	0.274	0.356	0.003
	PLAX	0.252	0.432	0.969
	PSSA	0.272	0.435	0.798

As can be seen in Figs. 6 and 7, the operator’s final choice of probe orientation and position was mostly in line with the statistical experience we obtained. This is clearly as expected since the operator was guided by a navigation system formed by statistical experience to operate the probe. However, we also noted that some of the operator’s choices were at the edge of the statistical range, which was due to the quick training of the operator on the characteristics of the ultrasound plane prior to the experiment, and therefore included some adjustments guided by subjective factors in their choices. To evaluate the quality of the acquired ultrasound images shown in Fig. 8, we mixed the acquired images with an equal number of standard-view images obtained by professional sonographers. Seven professional sonographers (three of whom have been in clinical practice for an average of three years and the other four for more than ten years) were invited to rate all of the images using the Likert scale where they can select an option on a scale from one to five (1: strongly disagree; 2: disagree; 3: neither agree nor disagree; 4: agree; 5: strongly agree). Specifically, there are three things that need to be scored in this study: 1. Whether this view contains all anatomical structures that the corresponding standard view should contain; 2. Whether each anatomical structure is clear and completely visible in this view; 3. Whether the content to be observed for diagnosing the related common diseases can be observed in this view. The average score results for each acquired view are shown in Table 5, and we take the average score results of the images from professional sonographers as the baseline for the standard view. Compared to the baseline, it can be demonstrated that the ultrasound views obtained by our navigation method are diagnostic.

Fig. 8.

Images acquired by non-clinical operators with no experience in ultrasound scanning: a A4C, b PLAX and c PSSA view. (Standard: illustrative diagram depicting the anatomical structures corresponding to the standard view, RV right ventricle, LV left ventricle, RA right atrium, LA left atrium, IVS interventricular septum, PPM posterior papillary muscle, APM anterior papillary muscle and AO aorta.)

Table 5.

The results of the average score for each acquired view and the baseline for standard-view

	View	Feature1	Feature2	Feature3
Case1	A4C	4.15	4.14	4.28
	PLAX	4.86	4.86	4.86
	PSSA	4.00	4.00	4.00
Case2	A4C	4.43	4.14	4.43
	PLAX	4.14	3.43	4.00
	PSSA	4.29	4.14	4.29
Case3	A4C	4.00	3.86	3.86
	PLAX	4.86	4.43	4.71
	PSSA	4.14	4.00	4.14
Baseline	A4C	4.29	3.81	4.38
	PLAX	4.76	4.43	4.67
	PSSA	4.67	4.14	4.52

Discussion

This work introduces a new approach to facilitate visual navigation for manual ultrasound scanning based on physician skill learning. The positioning information about the probe’s movement was recorded using vision-based methods. We then collected cardiac ultrasounds from 14 volunteers and analyzed the position and orientation of the ultrasound probe relative to the subject’s body by statistical methods. Furthermore, an AR-based navigation method was proposed to provide the operator with visual guidance to show the current location of the probe and indicate the expected location of the probe based on the previous statistical results. Ultimately, a validation experiment allowed an operator with no clinical experience or ultrasound scanning background to successfully obtain three cardiac ultrasound views of three volunteers with the support of the navigation system.

The camera and machine vision based tracking method given in this paper for probe motion is low-cost and easy to operate. In terms of real-time tracking accuracy, previous studies [15, 16] have shown that the average accuracy for translation and rotation of a single Aruco marker is approximately 3 mm and 3°, respectively. The ChArUco marker used in this study theoretically improves the accuracy as it is equipped with multiple Aruco markers. Our experimental results indicate that the probe position and orientation are within a range of tens of millimetres and tens of degrees. We are therefore confident that the proposed tracking method satisfactorily meets the requirements of the acquisition.

For the navigation system, the effectiveness of this navigation method is directly illustrated by the fact that a total of nine ultrasound views from three volunteers were effectively acquired by a non-clinical operator in the follow-up test. Such results are exciting because of their important potential value for both ultrasound promotion and training. Of course, there are some shortcomings in our existing work. First, because the data collection was conducted in the laboratory, the volunteers recruited for our collection and testing were all male. Second, the number of experts and volunteers invited to the collection process and the number of operators and subjects in the testing process are limited for practical reasons, but these issues can be improved by further follow-up testing. Naturally, as the collection database expands, the accuracy of navigation will be further improved. Finally, the current focus of skill learning and the proposed navigation method is primarily on the position and orientation of the ultrasound probe, without considering the contact force. One of the reasons is that the amount of applied force is not a relatively difficult skill to learn during training and it usually affects the contrast of the image which can be achieved through simple training of the operator. Of course, further research into contact force navigation will be carried out in our future study.

In addition, the proposed method of navigation based on prior knowledge and AR gives the expected position of the probe still as a range. The operator can try to adjust the probe to the center of the range, but it is also possible that the best location for a specific subject is not in the center of the predicted range. As with the choice of operator in the experiments, after receiving basic training on ultrasound view, the operator will certainly add subjective adjustments during actual operation, which should also be encouraged in real practice. The operators in this paper are completely non-clinical, while the intended users of the operating system are in fact primary care general practitioners or community physicians, for whom ultrasound training combined with the use of the navigation system can be expected to yield good results. Furthermore, the application of potential automated ultrasound view recognition algorithms [22, 23] could provide additional support for more accurate localisation of the standard view, which is our future development goal.

Although the study was based on the standard ultrasound views of the heart, the method can be transferred to the views of other organs, e.g., the liver and kidney. Our future work will focus on expanding the data and testing more participants. The ultimate goal is to deploy the navigation system in primary or community care units, allowing the operator to reduce the experience required for ultrasound scanning and to assist tiered care systems.

Author contributions

All authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by Mingrui Hao, Jun Guo, Cuicui Liu, Chen Chen and Shuangyi Wang. The first draft of the manuscript was written by Mingrui Hao and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Funding

This research was funded in part by the National Natural Science Foundation of China under Grant 62373352 and in part by the InnoHK program.

Declarations

Conflict of interest

The authors have no relevant financial or non-financial interests to disclose.

Ethical approval

This study was performed in line with the principles of the Declaration of Helsinki. Approval was granted by the Ethics Committee of Hangtian Center Hospital (2022/No.079).

Consent to participate

Informed consent was obtained from all individual participants included in the study.

Consent to publish

The authors affirm that human research participants provided informed consent for publication of the images in Figs. 1, 4 and 8.