Artificial Intelligence to Automatically Assess Scan Quality in Hip Ultrasound

Abstract

Purpose

Since it is fast, inexpensive and increasingly portable, ultrasound can be used for early detection of Developmental Dysplasia of the Hip (DDH) in infants at point-of-care. However, accurate interpretation\is highly dependent on scan quality. Poor-quality images lead to misdiagnosis, but inexperienced users may not even recognize the deficiencies in the images. Currently, users assess scan quality subjectively, based on image landmarks which are prone to human errors. Instead, we propose using Artificial Intelligence (AI) to automatically assess scan quality.

Methods

We trained separate Convolutional Neural Network (CNN) models to detect presence of each of four commonly used ultrasound landmarks in each hip image: straight horizontal iliac wing, labrum, os ischium and midportion of the femoral head. We used 100 3D ultrasound (3DUS) images for training and validated the technique on a set of 107 3DUS images also scored for landmarks by three non-expert readers and one expert radiologist.

Results

We got AI ≥ 85% accuracy for all four landmarks (ilium = 0.89, labrum = 0.94, os ischium = 0.85, femoral head = 0.98) as a binary classifier between adequate and inadequate scan quality. Our technique also showed excellent agreement with manual assessment in terms of Intraclass Correlation Coefficient (ICC) and Cohen’s kappa coefficient (K) for ilium (ICC = 0.81, K = 0.56), os ischium (ICC = 0.89, K = 0.63) and femoral head (ICC = 0.83, K = 0.66), and moderate to good agreement for labrum (ICC = 0.65, K = 0.33).

Conclusion

This new technique could ensure high scan quality and facilitate more widespread use of ultrasound in population screening of DDH.

Graphical abstract

Introduction

Developmental Dysplasia of Hip (DDH) occurs in 1–3 per 1000 live births [1] with large variations among ethnic groups [2]. DDH missed in childhood is a major risk factor for early onset hip osteoarthritis(OA) [2, 3] which is associated with a high economic burden worldwide [4, 5]. More than one-third of hip replacements in young adults (< 60 years) are due to undiagnosed DDH [6]. Conventionally, DDH is diagnosed by physical examinations including Barlow and Ortolani maneuvers, which have poor sensitivity beyond the neonatal period and miss cases of mild DDH [7–9]. Ultrasound imaging is more sensitive to mild DDH and is theoretically ideal for universal screening programs, as it is portable, inexpensive and safe from ionizing radiation (unlike X-rays). If DDH is diagnosed using ultrasound in early infancy (< 7 weeks), it can be treated noninvasively such as by Pavlik harness [10]. If missed, it may require multiple surgical procedures. Unfortunately, universal ultrasound screening is not recommended or performed in most countries due to factors including high user variability in assessment and costly false-positive scans [11]. An objective tool for assessment of scan quality could be helpful, especially if it can provide real-time feedback.

Traditional 2D ultrasound includes static image assessment by Graf criteria, relying primarily on measurement of the angle between ilium and acetabular roof (Alpha Angle; Fig. 1B). Alpha angles > 60° are considered normal and < 43 are considered severely dysplastic [12]. This requires a 2D scan in the Graf Standard Plane with clearly visualized landmarks such as the ilium, acetabular roof, labrum, and femoral head, which is difficult to acquire especially for novice sonographers. Slight variations in probe position risk incorrect diagnosis for two-thirds of neonates and half of infants evaluated [13].

Fig. 1.

A Ultrasound hip examination of an infant. Scanning was performed with a 3D linear array transducer (13VL5) along the coronal graf plane. B Ultrasound image of the hip with anatomical landmarks such as the ilium, labrum, femoral head and os ischium. Graf evaluation uses the alpha angle measured between the ilium and the acetabulum shown by the yellow dashed lines

3D ultrasound (3DUS) of the hips is easier for novice users to perform with minimal training [14], because the probe sweeps across the hip capturing hundreds of images, at least some of which are likely high quality. Computer aided diagnosis (CAD) techniques for diagnosing dysplasia from 3DUS have also shown significant improvements in reducing the need for follow-up scans [15–17].

However, in both 2DUS and 3DUS, scan quality is currently assessed manually and informally by the user at the time of scanning, a subjective process dependent on the experience of the user. In the absence of rigorous training, users might acquire inadequate images which could result in incorrect diagnosis and also significantly diminish the performance of CAD systems. Automatic techniques that assess scan quality in real time would help in eliminating such low-quality images upfront, advising a user to repeat a poor-quality scan while the patient is still present in the clinic.

Automatic assessment of ultrasound is challenging due to various factors such as the presence of shadowing, image artifacts that resemble anatomical structures and blurred image boundaries. This makes adapting conventional techniques, such as template-matching [18], shape-based methods [19], and feature-based methods [20, 21] to ultrasound difficult. With recent advances in Deep Learning(DL), data driven approaches using Convolutional Neural Networks (CNN) and recurrent neural networks (RNNs) [22] have been used for ultrasound plane detection. CAD systems focusing on scan quality assessment of the hip are less common, but have been proposed using CNNs [23] and RNNs [24].

In this paper, we develop a CNN-based technique to automatically assess hip ultrasound scan quality based on four landmarks from the Graf classification: iliac roof, labrum, os ischium, and femoral head. Our approach does not make any prior assumptions on data and uses separate CNNs for each imaging feature.

Materials and Methods

Ultrasound Scanning

In this study, we used a random subset of 207 scans originally obtained for a prospective study with institutional ethics committee approval allowing retrospective analysis of images for purposes of AI automation. In the study, 3DUS was performed on infants after written parental consent was obtained. Infants were aged between 26 and 183 days (68% female) and had been recommended for ultrasound based on clinical suspicion of DDH (due to risk factors such as hip laxity, asymmetrical skin creases, breech position, female sex, first-born status and ethnicity). Since DDH can be unilateral or bilateral, we included each hip separately, and in cases of normal hips we only included one hip per subject in the study.

During routine clinical examination, we scanned both hips using a 12 MHz linear transducer (L12-5; Philips Healthcare, Andover, MA), as per American College of Radiology recommendations [25]. We also acquired high-resolution coronal 3DUS images of each hip using a 13-MHz 3D linear array transducer (13VL5; Philips Healthcare, Andover, MA). 3DUS was captured with the transducer positioned with the head resting near the greater trochanter of the infant as shown in Fig. 1A. Sonographers were asked to acquire the image such that the central slice approximated the Graf standard plane. We performed a 3.2 s automated sweep through ± 15° to generate 256 slices, each 0.13-mm thick and containing 411 × 192 pixels measuring 0.11 × 0.20 mm.

Labeling of Training Data

The training set consisted of images from 100 3DUS scans with clinical gold-standard diagnosis based on radiologist and/or orthopedic surgeon assessment and 6 months clinical follow-up. We had 71 normal scans, 18 considered borderline requiring follow-up (typically Graf IIa), and 11 dysplastic cases that went on to treatment. Each image used in this study was assessed by a labeler (E.O) for scan quality using a scan quality scoring system [26] based on: the presence of labrum (class 0: absence, class 1: presence), midportion of the femoral head (class 0: absence, class 1: presence), os ischium (class 0: not visible, class 1: faintly visible, class 2: clearly visible), and iliac wing (class 0: not straight, class 1: straight but not horizontal, class 2: straight and horizontal). The labeler also marked key frames/slices in each image where the iliac wing, os ischium, labrum and femoral head were present. To avoid ambiguities in labeling we discarded ± 10 frames on either side of the keyframes.

CNN Architecture

We developed a CNN model consisting of convolutional layers and fully connected layers for classifying the image. The input 3DUS volume was analyzed frame-by-frame and resized to match the input shape of CNN as illustrated in Fig. 2. Separate networks were trained to predict each of the characteristics—flatness of iliac roof, visibility of labrum, os ischium, and mid portion of the femoral head. We used a brute force approach to train networks with varying numbers of convolutional layers and fully connected layers and selected the networks that gave highest accuracy.

Fig. 2.

Schematic diagram of the proposed AI approach for scan quality assessment. The 3DUS volume is analyzed frame-by-frame using four separate CNNs (CNN_ilium, CNN_labrum, CNN_os ischium and CNN_femoralHead) for each of the features. The models shown in the Fig represent the optimal configurations for each of the features

Sample Size

We expect the reliability of 5 readers (including AI as a 5th reader) to be better than the minimum acceptable reliability (ICC = 0.6). To measure a significant difference for expected ICC = 0.71, with significance level = 0.05 and power = 80% we need 101 samples. Allowing a dropout rate of 5% we used a total sample size 107.

Manual and AI Assessment of Scan Quality in Test Set

We used a test set of 107 3DUS images in different patients than the training set. In this data we compared the performance of the CNN against three human readers (reader 1, B.C, a radiology resident, reader 2 E.O, a graduate student in radiology, reader 3: A.H a research associate with experience in ultrasound hip image analysis). The readers scored the images online using Dataturks labeling software (www.dataturks.com) hosted on premise.

Ground Truth

All 107 test-set images were also scored by an expert—our lead radiologist, with fellowship training in pediatric and musculoskeletal radiology and 17 years of experience (JJ). While labeling, images were displayed in random order and all readers were blinded to clinical diagnosis and other patient information.

Statistics

We used Accuracy, Sensitivity, Specificity, negative predictive value (NPV) and positive predictive value (PPV) to compare the predictions of the CNN against ground-truth readings provided by the expert. We evaluated prediction for each criteria as a binary classification of adequate quality vs. inadequate quality.

Descriptive statistics of agreement of AI prediction with manual readings were also reported in terms of ICC (2,1) and Kappa. ICC values less than 0.5 indicate poor, 0.5–0.75 moderate, 0.75–0.9 good, and greater than 0.90 excellent reliability [27]. All statistics were computed using custom in-house software developed in Python 3.6 using sklearn and pingouin libraries.

Results

We trained several CNNs for categorizing images based on the quality of ilium, labrum, os ischium and femoral head. The performance of CNNs in scoring each 2D ultrasound frame in the validation set was evaluated in terms of accuracy (i.e. the proportion of images in which the CNN prediction matches exactly with the manual labels) as summarized in Table 1.

Table 1.

Accuracy of classification of CNNs in categorizing 2D frames based on the visibility of ilium, labrum, os ischium and femoral head on the validation set

Network	Validation set	Accuracy (%)
CNN_ilium	0-not straight: 380, 1-straight: 254, 2-straight and horizontal: 252	88
CNN_labrum	0-not visible: 128, 1-visible: 252	100
CNN_os ischium	0-not visible: 256, 1-faintly visible: 489, 2-clearly visible: 644	96
CNN_femoralHead	0-not visible: 128, 1-visible: 252	100

Since scanning was performed in a research setting by trained sonographers, low-quality images were less than typical representing an idealized scenario. Hence, we have tried to balance the validation set such that we have sufficient representation from each category. For example, there were only few images in which the labrum and femoral head was not visible, hence we used 128 negative samples (not visible) and 252 positive samples (visible).

We then compared the predictions of the CNNs against expert readings on 107 3DUS images by combining the predictions on the individual frames. In each category as a rule of thumb, images with at least 15% of frames continuously identified as class 1 or class 2 were categorized as adequate. Based on this heuristic our accuracy was ≥ 0.85 for os ischium, ilium, labrum,and femoral head. We also calculated the Sensitivity, Specificity, PPV and NPV for a test considering an inadequate-quality scan as a positive result, treating each network as a binary classifier with class 1 and class 2 images considered adequate and class 0 inadequate as summarized in Table 2.

Table 2.

Comparison of AI scoring of image landmark scan quality vs. human expert scores for ilium, labrum, os ischium and femoral head

	Accuracy	Sensitivity	Specificity	PPV	NPV
CNN_ilium	0.89	0.96	0.87	0.94	0.78
CNN_labrum	0.94	0.90	0.75	0.97	0.71
CNN_os ischium	0.85	0.86	0.81	0.91	0.72
CNN_femoralHead	0.98	0.99	0.66	0.99	0.66

To calculate the Sensitivity, Specificity, PPV and NPV class 1 and class 2 images were considered adequate for ilium and os ischium.

Examples of the automatic AI based quality assessment in representative images of varying quality are shown in Fig. 3. Figure 3A represents a low-quality image with none of the landmarks visible. The labrum and femoral head are clearly visible in Fig. 3B but the ilium is not straight or horizontal and the os ischium is only faintly visible. This image would be classified as moderate quality. Figure 3C shows an image of adequate diagnostic quality with all landmarks present and visible. In each of these images the AI assessment agreed with consensus of scores obtained from manual readings. Using our viewing software developed inhouse we overlaid the AI prediction on the original images for visual assessment and comparison.

Fig. 3.

Examples of images correctly identified by our technique. A shows a low-quality image in which none of the landmarks are visible. Figure 3B is of moderate quality C High quality image with all landmarks clearly visible. The predictions are overlaid on the images. In each case the highest quality is shown in green and lowest in red. For predictions of os ischium and ilium (B) scores of 1 are shown in yellow indicating moderate quality

There was variability in the manual scores assigned for some images. For example, the image shown in Fig. 4A ilium was scored as category 1 (straight but not horizontal) by 2 human readers and category 2 (straight and horizontal) by the other 2 readers. AI scored this image as category 1 (straight but not horizontal). Similarly, os ischium was scored as category 1 (faintly visible) by two readers and category 2 (clearly visible) by the other 2 readers for the image shown in Fig. 4B. AI scored this as category 2(clearly visible). In both cases the AI prediction would be correct in a binary sense, where both category 1 and category 2 are considered adequate and category 0 is inadequate.

Fig. 4.

Cases where interpretation of scan quality assessment can be subjective. A Ilium was scored as ‘straight but not horizontal’ by 2 readers and ‘straight and horizontal’ by the other 2 readers. B os ischium was scored as ‘faintly visible’ by 2 readers and ‘clearly visible’ by the other 2 readers

When we compared the agreement of CNN predictions against 4 human readers, AI predictions were treated as a fifth reader and compared with human readings (Table 3). The ICC values for all landmarks were higher when AI predictions were included. AI showed higher agreement with the expert than did the other 3 human readers in terms of kappa for ilium, os ischium and femoral head scoring.

Table 3.

Comparison of agreement of AI prediction with human readers and agreement among human readers without AI

Feature	Agreement of manual readers		Agreement of AI
	ICC (4 manual readers)	Kappa	ICC (4 manual readers + AI)	ICC (AI + Expert)	Kappa
Ilium	0.76 [0.70, 0.82]	0.55–0.74	0.81 [0.75, 0.84]	0.80	0.66
Labrum	0.51 [0.42, 0.58]	0.16–0.46	0.65 [0.56, 0.72]	0.61	0.33
os ischium	0.86 [0.80, 0.91]	0.6–0.75	0.89 [0.84, 0.92]	0.86	0.63
Femoral Head	0.77 [0.78, 0.88]	0.55–0.74	0.83[0.75, 0.88]	0.79	0.66

Note that ICC (2,1) is higher for all features when AI is included as a fifth reader (column 4) compared to the corresponding agreement between non-expert readers and expert (column 2)

Discussion

In this paper, we developed a technique to automatically evaluate hip dysplasia ultrasound scan quality based on the visibility of four landmarks in the images (ilium, labrum, femoral head, os ischium). As an automatic reader detecting poor-quality scans, our tool gave accuracy and reliability indistinguishable from human readers of varied experience. This tool could ultimately provide real-time feedback during hip ultrasound scanning, for user training and quality assurance.

Agreement between our AI technique and human readers ranged between moderate to high for anatomical landmarks, which was similar to the agreement between human readers. Agreement was good but not excellent for assessment of shape of the ilium and femoral head, which is not surprising as the concepts of the iliac wing forming a straight line and the femoral head a sphere are idealizations which do not hold exactly true, particularly in dysplastic hips. In all cases, inter-reader reliability on the quality score was higher when the AI technique was added as fifth reader, indicating high agreement with manual scoring. Similarly, agreement between AI and expert reading on the binary assessment of whether a scan was adequate vs. inadequate was within the range of values obtained from manual readings for all four landmarks.

A key feature of our technique is that we evaluate individual aspects of image quality separately based on image landmarks. Unlike end-to-end approaches that give a single score for overall image quality [24], our approach generates a more explainable framework using four CNNs. Our approach is more granular which reduces ambiguity in ground-truth labels as the expert makes a set of simpler (mostly binary) decisions. In routine clinical practice, 2DUS scans are analyzed using Graf assessment which requires a straight and horizontal ilium. Ideally measurements should be made at the deepest part of the joint where the diameter of the femoral head is highest and os ischium is clearly visible. Another important criteria for scan adequacy is visibility of the labrum for measuring the beta angle which is used in some centers for DDH assessment. During ultrasound examination the sonographer assesses these criteria before acquiring the image. However these judgements might be subjective resulting in inadequate scans. In 3DUS or their manual equivalent, cine ‘sweep’ videos obtained while the user ‘sweeps’ the probe across the hip, assessment of quality is tedious due to a large number of image slices. As a result, without rigorous training crucial anatomical landmarks could be missed. Our technique for automatic scan assessment can be used to flag inadequate 3D scans or 2DUS ‘sweeps’ during ultrasound examination.

Our automatic flagging technique can also be used as a pre-processing step in CAD systems. There are several CAD systems [15–17, 28–30] aimed at automatic interpretation of hip ultrasound. In general, these systems use image processing techniques for identifying the same image landmarks we use to assess quality (iliac roof, acetabulum, femoral head). Low image quality and inadequate visualization of these landmarks could potentially reduce the accuracy and performance of these systems. Our quality scoring tool could filter out these low-quality images and thereby improve the performance of CAD systems.

Our approach has limitations. Firstly, our ground-truth score is based on human assessment which is necessarily subjective. We noticed such ambiguity especially between scores of 1 and 2 for straightness of ilium and visibility of os ischium (Fig. 4). ‘Edge cases’ such as these are inevitable in qualitative scoring. Summing multiple features reduces the sensitivity of an overall score to ambiguity in individual features. Secondly, our validation data set was relatively small (n = 107) and, because it was performed in a rigorous research environment by experienced sonographers, contained fewer low-quality images than might be seen in a screening program. Severely dysplastic hips were also few in our dataset. So, the network might misclassify these as low-quality images. We plan larger studies where scans are performed by non-experts using handheld ultrasound devices, in which we expect more poor-quality images and more images of severely dysplastic hips for a more balanced training set. Thirdly, our approach is slice-based and does not consider temporal and spatial information of the 3D volume. 3DUS (as well as 2D ultrasound sweeps) have imaging artifacts caused by patient movement, accidental hand movement, shadowing or reverberation. Our technique does not yet specifically address these artifacts in a 3D scene, which might improve accuracy. Finally, this work was trained on 3DUS images, obtained using a costly dedicated 3D probe which is not in wide use. However, the image stacks obtained are computationally identical to cine ‘sweep’ video clips saved from universally available 2DUS linear probes, except for the strictly controlled inter-slice spacing in 3DUS. In future we will directly test the quality assessment network on 2DUS cine sweeps to confirm equivalent results.

This study shows that it is feasible to automatically distinguish high-quality hip dysplasia ultrasound images from non-diagnostic images. We envision incorporating this into a real-time feedback tool, so that a lightly-trained user would perform a quick 2DUS cine ‘sweep’ through an infant hip, and the AI would identify whether adequate images were obtained for diagnosis; if not, the user would re-scan the hip. This is much like a vending machine rejecting a dollar bill that is too crinkled. Then, a second AI network could examine the pre-filtered high-quality images to accurately diagnose hip dysplasia. This technology can either be used as an add-on in a clinical ultrasound machine or it can be used through a cloud based Machine Learning service to analyze images from a PACS system. It could be used as feedback to train novice users, and ultimately might render hip ultrasound sufficiently reliable and cost-effective to perform mass population screening.

Conclusion

We developed a new technique to automatically assess the image quality of infant hip ultrasound images based on the presence of imaging landmarks used in clinical practice. The new technique was highly accurate and showed high agreement with expert assessment. Our technique could be used as a real-time pre-processing step to help ensure a user has obtained diagnostic-quality images in a training environment and in population hip dysplasia screening.

Declarations

Conflict of interest

Dornoosh Zonoobi and Jacob L. Jaremko are co-founders of MEDO.ai Inc, which is a company that develops AI solutions for ultrasound. The remaining authors declare that they have no conflict of interest.

Research involving human participants

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee (include name of committee + reference number) and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Informed consent

Informed consent was obtained from all individual participants included in the study.