Validation of an artificial intelligence-based method to automate Cobb angle measurement on spinal radiographs of children with adolescent idiopathic scoliosis

Abstract

BACKGROUND

Accurately measuring the Cobb angle on radiographs is crucial for diagnosis and treatment decisions for adolescent idiopathic scoliosis (AIS). However, manual Cobb angle measurement is time-consuming and subject to measurement variation, especially for inexperienced clinicians.

AIM

This study aimed to validate a novel artificial-intelligence-based (AI) algorithm that automatically measures the Cobb angle on radiographs.

DESIGN

This is a retrospective cross-sectional study.

SETTING

The population of patients attended the Stollery Children’s Hospital in Alberta, Canada.

POPULATION

Children who: 1) were diagnosed with AIS, 2) were aged between 10 and 18 years old, 3) had no prior surgery, and 4) had a radiograph out of brace, were enrolled.

METHODS

A total of 330 spinal radiographs were used. Among those, 130 were used for AI model development and 200 were used for measurement validation. Automatic Cobb angle measurements were validated by comparing them with manual ones measured by a rater with 20+ years of experience. Analysis was performed using the standard error of measurement (SEM), inter-method intraclass correlation coefficient (ICC_2,1), and percentage of measurements within clinical acceptance (≤5°). Subgroup analysis was conducted by severity, region, and X-ray system to identify any systematic biases.

RESULTS

The AI method detected 346 of 352 manually measured curves (mean±standard deviation: 24.7±9.5°), achieving 91% (316/346) of measurements within clinical acceptance. Excellent reliability was obtained with 0.92 ICC and 0.79° SEM. Comparable performance was found throughout all subgroups, and no systematic biases in performance affecting any subgroup were discovered. The algorithm measured each radiograph approximately 18s on average which is slightly faster than the estimated measurement time of an experienced rater. Radiographs taken by the EOS X-ray system were measured more quickly on average than those taken by a conventional digital X-ray system (10s vs. 26s).

CONCLUSIONS

An AI-based algorithm was developed to measure the Cobb angle automatically on radiographs and yielded reliable measurements quickly. The algorithm provides detailed images on how the angles were measured, providing interpretability that can give clinicians confidence in the measurements.

CLINICAL REHABILITATION IMPACT

Employing the algorithm in practice could streamline clinical workflow and optimize measurement accuracy and speed in order to inform AIS treatment decisions.

European Journal of Physical and Rehabilitation Medicine 2023 August;59(4):535-42

DOI: 10.23736/S1973-9087.23.08091-7

© 2023 THE AUTHORS

ORIGINAL ARTICLE

NEW PERSPECTIVES IN SCOLIOSIS AND SPINAL DEFORMITIES: AN UPDATE FROM THE 2023 ANNUAL SOSORT MEETING

(Cite this article as: Wong JC, Reformat MZ, Parent EC, Stampe KP, Southon Hryniuk SC, Lou EH. Validation of an artificial intelligence-based method to automate Cobb angle measurement on spinal radiographs of children with adolescent idiopathic scoliosis. Eur J Phys Rehabil Med 2023;59:535-42. DOI: 10.23736/S1973-9087.23.08091-7)

Adolescent idiopathic scoliosis (AIS) is a three-dimensional spinal disorder that is characterized by lateral curvature coupled with axial vertebral rotation. The disorder affects 1-3% of adolescents and can result in cardiopulmonary compromise and back pain if left untreated.¹ The gold standard for quantifying AIS severity for diagnosis and monitoring is the Cobb angle measurement method on a posteroanterior (PA) radiograph. This consists of identifying the upper endplate of the superior most tilted vertebra and the lower endplate of the inferior most opposing tilted vertebra.² The sum of these two angles is known as the Cobb angle and measuring it is crucial for determining the appropriate treatment options for a child with AIS.³ Therefore, it is essential that the Cobb angle is measured accurately and reliably. Manual Cobb angle measurement suffers from 3-5° intra-observer and 5-7° inter-observer variation.⁴ However, these values are reported for experienced raters, and so the variation could be higher for less experienced raters.

Because clinicians see a large volume of children with AIS, they may seek an automated method of Cobb angle measurement to reduce clinical work load.⁵ However, clinicians may be wary of implementing automated methods of diagnosis due to concern regarding black box diagnosis or medical liability due to potential algorithmic errors.⁶ Therefore, an essential step for knowledge translation of an automatic measurement method is provision of some interpretability, or insight into how the algorithm came to its output. By accomplishing this, clinicians can intervene when the algorithm has provided invalid measurements.

Other research groups have developed their own automatic Cobb angle measurement methods for PA radiographs to improve measurement reliability and streamline clinical workflow. Many of them have incorporated artificial intelligence (AI), specifically convolutional neural networks (CNN), into their methods. Fu et al. developed a CNN-based algorithm, where one CNN located the four corners of each vertebral body and another determined the Cobb angle from the predicted corners. They achieved a mean absolute difference (MAD) and standard deviation of absolute differences (SD) of 3.2±2.1° between their automatic and manual measurements. However, their algorithm provides no explanation for how it calculates the Cobb angle from the corner landmarks, meaning the interpretability criterion is not fulfilled.⁷ Horng et al. also used a CNN to automatically label the individual vertebral bodies and measure the Cobb angle. The MAD±SD between their automatic and two sets of manual measurements was 3.0±2.0° and 2.5±1.7°. Their algorithm was interpretable, but their test set lacked curve severity variety, as only Cobb angles below 20° were included in their test set.⁸ Treatment options for AIS past regular observational monitoring start to be considered for curves past 20°, meaning that their algorithm is not clinically feasible until accurate measurements are obtained for more severe curves. Zhao et al. took a similar approach of segmenting the individual vertebral bodies using a CNN. They achieved a MAD of 2.5° between automatic and manual measurements, but they do not provide Cobb angle distribution information on this test set, so it is unknown whether the algorithm measures accurately on a wide range of curve severities.⁹ There are other researchers that have published their own automatic measurement methods.^10-14 However, the accuracy of their automatic measurements is poor, ranging from a 3.3° to 6.1° MAD, making them clinically infeasible.

In the literature, there is no completely automatic method for Cobb angle measurement that measures accurately and reliably on a wide range of curve severities, while providing algorithmic outputs that offer interpretability for the clinicians. Consequently, this study aimed to validate a novel AI-based algorithm that automatically measures the Cobb angle on PA radiographs, in a wide range of curve severities, and to evaluate its clinical feasibility in terms of measurement time and interpretability.

Materials and methods

Patient population

A total of 330 standing spinal PA radiographs of children with AIS were extracted retrospectively from the Stollery Children’s hospital in this cross-sectional observational study. The images had an average width of 1993 pixels and height of 4143 pixels. Of these 330, 130 images were used for AI model training, with data augmentation of random rotation, zooming, translation, and horizontal flipping to increase the number of images to >100,000. All images were labelled using a segmentation graphical user interface in MATLAB. The other 200 images were used for automatic Cobb angle measurement validation. The common inclusion criteria for both training and validation data were children who: 1) were diagnosed with AIS; 2) were aged between 10 and 18 years old; 3) had no prior surgery; and 4) had a radiograph out of brace. Based on these criteria, 130 images were chosen for CNN training, as that number was deemed sufficient to train CNNs that produce accurate spinal feature segmentations. For the 130-image training set, we chose not to exclude images based on curve size to improve the clinical robustness of the AI model, and so these images were selected specifically to include a wide range of curve severities and different spinal shapes. For the validation set, we had additional inclusion criteria of only selecting radiographs that had a manually measured major Cobb angle of 10-55°, as these represent non-operative cases at our scoliosis clinic, and radiographs that were not included in the 130-image training set. Once the eligible population of validation radiographs was determined, the 200 validation images were selected randomly without looking at the subjects’ radiographs or Cobb angles. While subjects with a manually measured major curve below 10° were excluded, any manually measured minor curves that were below 10° were included in the validation comparison, provided that the subject has a major curve above 10°. The chart review ethics approval was granted by the University of Alberta health research ethics board.

Each participant was imaged with either a digital full-spine conventional or EOS X-ray system (including low-dose and micro-dose radiographs). Both training and validation sets were split between half conventional and half EOS radiographs to ensure no inherent bias towards one system. Examples of conventional and EOS radiographs are illustrated in Figure 1. Within the 130-image set for AI model training, the population comprised of 21 male and 109 female participants who had an average Cobb angle of 24.6±12.1° (including all major and minor curves with range: 6-97°) and an average age of 14.3±1.8 years.

Figure 1.

—Examples of conventional (A) and EOS (B) radiographs.

AI-based automatic measurement method

Using the 130-image training set, two CNNs were developed: one to segment the spinal column, and the other to segment individual vertebral bodies. Using these two CNNs in a cascaded fashion allowed us to localize the vertebral bodies and derive their bounding box tilt angles to automatically measure the Cobb angle. The overall procedure of the AI algorithm is shown in Figure 2. The algorithm is an improved version of our previous work,¹⁵ adding vertebral body image registration to improve measurement reliability.

Figure 2.

—Procedure of automatic Cobb angle measurement performed by the AI algorithm.

Both CNNs were trained using images of spinal columns and vertebral bodies labelled by the primary author, who had over three years of scoliosis research experience. A senior co-author with over 20 years of scoliosis research experience confirmed the labels on ten images for each segmentation task before the primary author proceeded with the remaining images.

When the Cobb angles are measured, the AI algorithm also outputs the segmentations, which are used to derive the automatic Cobb angle measurements, overlaid on top of the initial input image. This illustrates how the algorithm measures the Cobb angles so that the user can verify the result and decide if any measurements need to be adjusted.

Validation

Evaluating the performance of the developed algorithm consisted of comparing automatic Cobb angle measurements with the manual ones in the 200-image validation set. Both major and minor curves were considered in this comparison. Automatic measurements were not manually adjusted in any way, even after seeing the algorithm’s segmentation outputs. Using the method of Cobb et al.,² manual measurements were performed by the same senior co-author who verified annotations before CNN training. He was blinded to the automatic measurements. From this automatic-manual measurement comparison, the MAD, SD, inter-method intraclass correlation coefficient (ICC_2,1), and standard error of measurement (SEM) were calculated. The ICC_2,1 was qualitatively evaluated according to Koo’s definitions of poor (<0.5), moderate (0.5-0.75), good (0.75-0.90), and excellent (≥0.90).¹⁶ Additionally, the percentage of measurements within clinical acceptance was derived as a measure of accuracy. Clinical acceptance was defined as when an automatic measurement was within at most 5° of the paired manual measurement. This threshold is frequently used due to intra-observer and inter-observer manual Cobb angle measurement variation.⁴ Bland-Altman analysis was also conducted to visualize the measurement comparison and identify any potential systematic algorithmic errors.¹⁷

Results were further analyzed by curve region, curve severity, and X-ray system to identify any potential systematic biases in the algorithm. Curve region was separated according to the location of the apex with upper thoracic (UT) being the apex within T2-T6, main thoracic (MT) T7-T11, thoracolumbar (TL) T12-L1, and lumbar (L) L2-L4.¹⁸ Curve severity was separated into three groups: mild (<25°), moderate (25-45°), and severe (≥45°). These thresholds were chosen because they roughly correspond to when clinicians consider the different treatment options of observation, bracing, and surgery, respectively.¹ ICC_2,1 values were not reported for the curve severity groups because of the risk of attenuation that comes with restricting the population variance. In terms of X-ray system, analysis was separated by conventional versus EOS. Independent samples Student’s two-tailed t-tests were performed to determine if the MADs of automatic versus manual measurements were significantly different between subgroups in the same category (such as mild vs. moderate).

Measurement time per radiograph was recorded, starting when the image was input into the algorithm and ending when the Cobb angle measurements were output. An independent samples Student’s one-tailed t-test was also conducted to evaluate if radiographs performed by one X-ray system were measured more quickly than the other. A P value less than 0.05 indicated statistical significance. All statistical analysis was performed using the Python libraries, pingouin and pandas. All measurements were run on a Windows computer with an i7-12700 Intel CPU and an NVIDIA GeForce RTX 3060 Ti GPU.

Results

Accuracy and reliability

There were 352 curves that were manually identified and measured in the 200-image validation set. There were 22 male and 178 female participants and none of the participants had missing data. Their average Cobb angle was 24.6±9.7° (range: 8-52°) with an average age of 14.1±1.8 years. The automatic algorithm detected 346 (98%) of these manually measured curves. Only 6 curves were missed, and these curves were all mild, ranging from 10-21°. The average Cobb angles of the 346 curves were 24.7±9.5° and 26.0±10.5° for manual and automatic measurements, respectively. The method achieved 91% of measurements within clinical acceptance with a 2.8° MAD, 0.92 ICC_2,1, and a 0.79° SEM. The full results with subgroup analysis are listed in Table I. No statistically significant differences were found in the MADs between the pairwise comparisons within each group of analysis (severity, region, and X-ray system). In other words, no significant difference in performance was observed between the different pairs of subgroups examined, suggesting the AI method performed consistently across different categories. The P values for all pairwise comparisons are listed in Table II.

Table I. —Automatic vs. manual Cobb angle paired measurement comparison results.

Grouping	nm	nmiss	μm±σm	μa±σa	MAD±SD	%clinical (nw/np)	SEM	ICC2,1 [95% CI]
All	352	6	24.7±9.5°	26.0±10.5°	2.8±2.8°	91% (316/346)	0.79°	0.92 [0.91-0.94]
Mild	192	6	17.4±4.1°	18.6±5.3°	2.7±2.4°	93% (173/186)	1.29°	--
Moderate	148	0	32.0±5.2°	33.5±7.1°	3.0±3.3°	89% (131/148)	1.67°	--
Severe	12	0	48.0±2.2°	49.4±3.8°	2.6±1.5°	100% (12/12)	0.97°	--
Upper thoracic	38	1	22.6±8.8°	23.2±10.1°	2.9±1.9°	95% (35/37)	0.49°	0.93 [0.87-0.96]
Main thoracic	152	2	26.7±10.3°	28.4±11.6°	2.9±3.4°	90% (135/150)	0.99°	0.92 [0.90-0.95]
Thoracolumbar	45	1	26.9±10.6°	28.3±10.1°	2.6±1.8°	95% (42/44)	0.38°	0.95 [0.93-0.98]
Lumbar	117	2	21.8±7.3°	22.9±8.1°	2.8±2.5°	90% (104/115)	0.85°	0.88 [0.85-0.92]
Conventional	175	3	23.8±10.1°	25.0±10.9°	2.9±2.5°	91% (157/172)	0.63°	0.94 [0.92-0.96]
EOS	177	3	25.5±8.9°	27.0±10.0°	2.7±3.1°	91% (159/174)	0.97°	0.90 [0.89-0.94]

n_m: number of curves manually measured; n_miss: number of curves missed by AI algorithm; μ: mean of Cobb angle measurements; σ: standard deviation of Cobb angle measurements; _m: manual; _a: automatic; MAD: mean absolute difference; SD: standard deviation of absolute differences; %_clinical: percentage within clinical acceptance (≥5°); n_w: number of curves within clinical acceptance; n_p: number of paired manual-automatic measurements; SEM: standard error of measurement; ICC_2,1: inter-method intraclass correlation coefficient; 95% CI: 95% confidence interval.

Table II. —Student’s t-test P values for MAD comparisons between different subgroups.

Comparison	P
Mild vs. moderate	0.24
Mild vs. severe	0.98
Moderate vs. severe	0.44
Upper thoracic vs. main thoracic	0.93
Upper thoracic vs. thoracolumbar	0.45
Upper thoracic vs. lumbar	0.79
Main thoracic vs. thoracolumbar	0.48
Main thoracic vs. lumbar	0.85
Thoracolumbar vs. lumbar	0.56
Conventional vs. EOS	0.77

P<0.05 indicates statistical significance.

Based on Bland-Altman analysis, the AI algorithm overestimated the Cobb angle with a bias of 1.3° and limits of agreement of (-6.0°, 8.7°). The bias is significant, as the equality value 0 was not contained within its 95% confidence interval (0.94°, 1.74°). Figure 3 illustrates a Bland-Altman plot, color-coded by region. From the plot, the errors are relatively uniform throughout the whole range of curve severity.

Figure 3.

—Bland-Altman plot of automatic vs. manual Cobb angle measurements, color-coded by curve region, with bias (black line) and limits of agreement (red lines).

Measurement time and interpretability

The AI algorithm took on average 18±10s to measure the Cobb angles per radiograph. Radiographs taken by the conventional system (26±8s) took significantly longer on average compared to the EOS system (10±3s) (P<0.05). The estimated time for an experienced rater to manually measure the Cobb angles on a child with AIS is 30s, meaning that our algorithm is on average quicker than an experienced rater, even for conventional radiographs. Examples of measurement and segmentation outputs are illustrated in Figure 4.

Figure 4.

—Clinically acceptable measurement outputs from the AI algorithm, demonstrating the interpretability that it offers, with green boxes indicating relevant vertebrae used in the measurement and cyan text indicating a Cobb angle value.

Discussion

Results analysis

The AI algorithm achieved a 91% clinical acceptance rate and a SEM of 0.79° for all measurements. Additionally, the algorithm measured radiographs within only 18s on average. This high accuracy and quick measurement time, combined with the interpretable outputs that illustrate how the Cobb angles were measured, demonstrate the clinical feasibility that the proposed algorithm offers. No statistically significant differences were found in the MADs within each subgroup comparison, indicating that the proposed algorithm contained no systematic biases towards measuring certain curves or images less accurately than others in terms of curve severity, curve type, and X-ray system. However, the algorithm tended to overestimate the true curve severity by 1.3°. While it was statistically significant, the value of the bias is small relative to the clinical acceptance threshold, and there was no evident reason for systematic overestimation found upon further investigation of the segmentations.

Conventional radiographs took a statistically significant longer time to measure than EOS radiographs, despite there being no statistically significant difference in measurement performance between the two. This is likely due to the overall poorer image quality in the conventional radiographs that can give the algorithm greater difficulty in localizing the individual vertebral bodies. However, the algorithm can overcome this poorer quality and still achieve comparable measurement accuracies at the expense of higher computation time.

Table III lists a comparison of our results with other methods in the literature that were found to be competitive with our method in terms of measurement accuracy.^7-9 Horng et al.⁸ and Zhao et al.⁹ reported lower MADs than the current method, but Horng et al.⁸ tested only on mild curves <20° and Zhao et al.⁹ did not specify its curve severity distribution. Fu et al.⁷ did have a wide range of curve severity; however, the MAD value was larger than the current method. While their MAD was within the accepted clinical measurement variability threshold, Fu et al. did not provide interpretable results. Therefore, the current method performed the most accurately while balancing a wider range of curve severity and still providing interpretability.

Table III. —Comparison of Cobb angle measurement performance on validation sets for various automatic algorithms.^7-9.

Method	MAD±SD	Curve distribution	Number of validation images	Interpretable
Fu et al.7	3.2±3.1°	2-92°	240	No
Horng et al.8	2.5±1.7°	<20°	35	Yes
Zhao et al.9	2.5°	--	75	Yes
Current method	2.8±2.8°	24.6±9.7° (8-52°)	200	Yes

Limitations of the study

One limitation of the study is the scope of the validation set population, namely the inclusion of only participants diagnosed with AIS and the low number of severe curves. We have no reason to believe that our method would incorrectly diagnose non-scoliotic curves as scoliotic, but nevertheless, non-scoliotic cases should be tested to officially confirm that the method will identify them as healthy. As for the number of severe curves, there were only 12 present, since there was an inclusion criterion of having a major Cobb angle <55°. While it is promising that the algorithm measured all 12 of these curves within clinical acceptance, further validation would need to be performed on more curves ≥45° to prove complete clinical feasibility. Additionally, the method currently uses the bounding boxes of the vertebral body segmentations to derive their tilt angles, but this differs from the manual method where the tilts of the upper and lower endplates of the vertebrae are employed. This difference in measurement method may lead to higher inaccuracies in cases where severely wedged vertebrae are directly used for measurement. However, severe wedging is more common in the apical vertebra of the curve, meaning that these potentially problematic cases are rarer.¹⁹ Nevertheless, further analysis on the effect that severe wedging has on measurement accuracy should be conducted with future validation on a test set with more severe curves.

Another limitation involved the scope of the imaging systems and patient posture of the radiographs investigated. Two different X-ray systems were tested in this study, which is an improvement over the other existing competing methods.^7-9 Because spinal radiographs taken by the EOS system are imaged in a more standardized procedure and protocol, the developed algorithm should be accurate on EOS radiographs from other centers without requiring further validation. However, radiographs taken by a conventional system may vary more, especially with regards to image quality or if stitching was involved. Therefore, the developed algorithm may need further validation before applying it to these centers. As for patient posture, only radiographs taken in a standing position were tested. However, other types of radiographs can be imaged for AIS in terms of posture, including supine or bending, or purpose based on treatment, including in-brace or post-operative. Future work should involve expanding the algorithm to accurately measure all these different types of radiographs. It would also be worth exploring using machine learning to automatically measure the Risser sign.

Finally, manual measurement time was only estimated by our experienced rater, and not measured directly. Therefore, we cannot analyze with statistical significance whether automatic measurement was quicker than manual measurement by an experienced rater.

Conclusions

An AI-based algorithm for automatic Cobb angle measurement of children with AIS was developed and validated on a 200-image set of PA radiographs. The method achieved a high accuracy, with 91% of measurements within clinical acceptance (≤5°), and excellent reliability, with a SEM of 0.79°. The algorithm performed comparably within curve severity, curve region, and X-ray system, and within each of these groups of analysis, no statistically significant differences in measurement performance were found between the subgroups. The average Cobb angle measurement time per radiograph was 18s±10s, which is slightly faster than manual measurement by an experienced rater. Furthermore, the algorithm displays the spinal segmentation on radiographs, allowing clinicians to quickly verify measurements, thereby fulfilling a key aspect required for knowledge translation. Future work consists of validating on more severe curves and on different X-ray systems from other centers to ensure comparable measurement performance. Once completed, this algorithm could streamline clinical workflow and offer robust measurements to inform AIS treatment decisions.