Automatically measuring the Cobb angle and screening for scoliosis on chest radiograph with a novel artificial intelligence method

Abstract

Objectives: To establish an automatic approach for the measurement of the Cobb angle and the diagnosis of scoliosis on chest radiograph. Methods: We developed an artificial intelligence (AI) automatic program which contained a supervised learning module and an inference module. After the filtering and pre-processing process, 96 images from the Shenzhen chest X-ray set were used for training with the supervised learning module, and 491 test images were separately gauged by the AI and the corresponding manual methods. The results of the two methods were further compared through statistical analyses. Results: Among the test images, 6068 (99.49%) vertebral bodies were identified within the deviation of one vertebral segment. The value difference between the Cobb angle obtained by the AI program and that measured by specialists was 0.4020±0.8703. The intraclass correlation coefficient of 0.915 indicated the strong agreement. AI scoliosis diagnosis achieved an accuracy of 98.37%, with a specificity of 98.73%, a sensitivity of 88.24% and a kappa coefficient of 0.781. And the area under the receiver operating characteristic curve of 0.979 confirmed the consistency of the two methods in diagnosis. Conclusions: We developed a novel automatic AI method with the abilities to measure the Cobb angle, and to identify the approximate vertebral segment and diagnosis of scoliosis on chest radiograph. The results suggest that this method might be a promising alternative strategy for scoliosis screening on chest radiograph and worth further investigation.

Introduction

As a three-dimensional structural spinal deformity, scoliosis is characterized by raised ribs, unbalanced shoulders, and an asymmetrical waist. Even with the development of medical imaging technology, which leads to the replacement of two-dimensional images (X-ray) with three-dimensional images (CT or MR) in many medical fields, X-ray still plays an important role in the diagnosis and evaluation of scoliosis. Measuring the Cobb angle, the degree of spinal curvature on the posteroanterior or anteroposterior spinal X-ray, is routinely used for assessing the severity of scoliosis [1]. The angle between the two tangents of the cephalad end vertebra’s superior surface and the caudad end vertebra’s inferior surface on the coronal plane is not completely equivalent to the real spinal deformity. Although some alternative methods have been proposed [2-4], the Cobb angle is still considered as the gold standard in current clinical diagnosis. When the lateral curvature is over 10°, the diagnosis of scoliosis should be confirmed.

As an essential for selecting therapeutic methods, monitoring spine deformities and evaluating therapeutic effects, the measurement of the Cobb angle should be accurate enough to represent the actual state of the spinal structure. However, manual measurement is faced with the problem of reliability and reproducibility. The measurement errors of the same curve from intra- and interobserver measurements have been reported to vary between 3-7° [5,6]. And the observer’s subjectivity and experience are related to these variations [6].

Measuring the Cobb angle with the manual method is labor-intensive and time-consuming. As artificial intelligence (AI) technologies become increasingly sophisticated, there is growing hope that the computer will take over some human work such as the measurement of Cobb angle or the diagnosis of scoliosis one day. Compared with manual work, the result carried out by computer usually shows better reproducibility and higher efficiency. Consequently, it seems promising to solve the problem of reliability and reproducibility in the measurement of Cobb angle with artificial intelligence. Multiple previous studies have tried to measure the Cobb angle or diagnose scoliosis by computer programs [7-11]. But much more attention needs to be paid due to the complicated clinical environment, which makes measurement of the Cobb angle automatically a long way to go.

In this paper, we present a novel AI method with the ability to measure the Cobb angle, identify vertebral segments and screen for scoliosis on chest radiograph. It processes by automatically simulating the manual procedure in dealing with chest radiography.

Methods and materials

Dataset

This study used an open-source chest X-ray (CXR) image dataset published by the National Library of Medicine, National Institutes of Health, Bethesda, MD, USA [12,13]. The chest X-rays, acquired from outpatient clinics of Shenzhen No. 3 People’s Hospital in China, were collected as part of the daily out-patient routine within one month. All personal information was deidentified, and the use of these images had been exempted by the Ethics boards [12]. There were altogether 662 frontal chest x-ray images, captured by Philips DR Digital Diagnose system, namely 326 normal and 336 pulmonary tuberculosis x-rays. No case of spinal tuberculosis, spinal tumor or spinal fracture was observed in this image dataset.

The inclusion criterion for this study was the chest X-ray image from people aged between 10 and 60. And the exclusion criteria were as follows: (I) CXRs with interference from extracorporeal objects; (II) CXRs with spinal implants. Among the eligible 589 images, 96 were randomly selected for model training or model validation, and the other 493 were used as the test set.

Image pre-processing

The pre-processing procedure contained image enhancement and size standardization. The spinal structure was not the original highlight of CXR. It could be vague when overlaid with the ribcage or inner organs such as the heart and lung [14]. Therefore, the gamma transformation was applied to suppress noises derived from soft tissues, adjust gray levels and stretch the contrast of the spinal structure. The function formula and function graph were represented in Figure 1B. As the gray-scale histograms shown in Figure 1A and 1C, the contrast of the original brighter pixels was stretched and that of the opposite pixels was suppressed. After this transformation, the spinal details were stressed, making it much easier for further annotation and detection.

Figure 1.

A. The gray-scale histograms of the original Chest X-ray image. The x-axis demonstrates gray scale values from 0 to 255. The y-axis shows the number of pixels. B. The gamma transformation’s function graph. The x-axis and y-axis separately demonstrate the gray scale values before and after the gamma transformation. C. The gray-scale histograms of the image after the gamma transformation. The x-axis demonstrates gray scale values and the y-axis displays the number of pixels. D. The original Chest X-ray image. E. The image after the pre-processing procedure.

For original images from the Shenzhen set, width and high varied slightly around 3000 pixels [12]. To modify the shape to square, each rectangular image was padded with black margins. Then padded images were resized to 1024 × 1024 pixels. All these pre-processing procedures were undertaken by a simple batch script written in Python.

Image annotation

For each CXR in the training set and validation set, the manual annotation was performed from the first thoracic vertebra to the bottom of the image. Bounding boxes and segmentation edges of vertebral bodies were annotated with the mouse control. If the inferior endplate of the lowest vertebral body segment was not completely displayed, this segment would be ignored. On the basis of segmentation annotation, the parallel auxiliary line and 13 critical points of each endplate were further annotated. During the annotation process, for indistinct structures, the approximate positions were estimated according to the relationship with pedicles and adjacent vertebral bodies. An experienced spine surgeon carried out this process, then another senior surgeon checked and revised the results. Afterwards, the annotated data was packaged into files as the ground truth.

AI program

The main program, developed in Python, contained a supervised learning module and an inference module. The supervised learning module could be used independently for model training or together with the inference module as the gauging program.

Supervised learning

The detection and segmentation tasks of our project were mostly based on mask region-convolutional neural network (Mask R-CNN), a flexible, robust and general framework [15]. The latter part of the Mask R-CNN framework was slightly modified, and an endplate critical point detection branch was added into the network. For each detected object, namely the detected vertebral body, 26 critical points were designed for the description of superior and inferior endplates. At the beginning of model training, pretrained weights derived from Microsoft COCO dataset (https://cocodataset.org/) were loaded into the backbone and region proposal layer. Images and annotations from the training set and validation set were then fed into the network. After a series of training and filtering, we got the final trained weight.

Inference process

At the testing stage, the trained weight was loaded into the supervised learning neural network in advance, and test images from the test set were pre-processed and fed into the supervised learning module. Then the outputs were input to the next inference engine. Within the inference process, bounding boxes with abnormal locations were eliminated according to the positional relationship, and the segment name of each vertebral body was obtained based on the sequence of corrected bounding boxes. For the vertebral body at the bottom of each CXR, the integrity was uncontrollable, which might seriously interfere with the fitting of the vertebral endplate in the test. Therefore, the lowest vertebral body of each image was designed to be ignored.

Then locations of bounding boxes were used to fit the central spinal curve with sixth order polynomial curves, and endplate fitted straight lines were obtained by the least square method on the basis of superior and inferior endplates’ critical points. After a series of inference, these fitted straight lines were synthetically corrected with the contour of vertebral bodies, the central curve and slopes of the adjacent endplates. Then slope angles of endplates were successively checked from top to bottom. For those outliers, the inference engine would make a further comprehensive correction that combined with critical points, the central line, instance masks’ contours and other factors. Intersection angles between the upper vertebra’s superior endplate and lower vertebra’s inferior endplate were automatically calculated and compared in sequence. The largest one was then chosen as the Cobb angle of the CXR. When the Cobb angle was 10° or higher, the patient would be diagnosed with scoliosis by the AI program.

Manual measurement

For each test image that could pass through the AI test, a semi-automatic Python script with graphical user interface was used to determine the Cobb angles. Similar to the traditional method, the most deviated cephalad end and caudad end vertebrae were firstly determined by the eye, and then the parallel straight lines of the corresponding superior and inferior endplates were marked off with the mouse control. The marking process was produced by an experienced spine surgeon, and then another senior surgeon reviewed and calibrated the marked endplate lines. Afterwards, the manual measurement result of the Cobb angle was figured out by computer.

Statistical analysis

Statistical evaluations were carried out with the assistance of SPSS version 26. A P value < 0.05 was regarded as statistically significant. The intraclass correlation coefficient (ICC) was used to check the agreement between the Cobb angle obtained by AI and that measured by manual. McNemar’s test was used to determine whether there was any difference between the AI and manual diagnoses of scoliosis. Kappa coefficient was used to evaluate the agreement between the two methods. And receiver operating characteristic (ROC) curve was used to assess the diagnostic ability of the AI program.

Results

In this study, all images were used after pre-processing. Compared with the original CXR such as Figure 1D, the image after the gamma transformation (Figure 1E) showed more details of the bony structure, and noises derived from soft tissues were suppressed to some extent. And all input images were padded and scaled to the same size (1024 × 1024 pixels) in the batch processing stage.

The automatic detection results of the supervised learning and inference process were shown in Figure 2. Auxiliary lines of superior and inferior endplates, fitted on the basis of a combination of bounding boxes (Figure 2B), the central curve (Figure 2C), critical points (Figure 2D), vertebral masks (Figure 2E) and so on, were separately displayed with blue color and pink color (Figure 2A). For each image, the Cobb angle was automatically calculated, and the most deviated cephalad end and caudad end vertebrae were framed with purple and green boxes separately. Moreover, the estimated segment names were printed near the corresponding vertebrae (Figure 2F). Under the hardware environment of NVIDIA Titan X GPU and Intel Core i5 CPU, the execution speed of the AI test program is around 3.0 seconds per CXR. Among the test set, only two images’ detections failed due to the detection of mistaken bounding boxes. And these two images were eliminated from the subsequent statistics.

Figure 2.

A. Overview image of the automatic detection. B. The visualization result of the detected vertebral bounding boxes and corresponding vertebral segment names. C. The visualization result of the fitted spinal central curve. D. The visualization result of the endplates’ critical points. Each color represents a type of critical point. E. The visualization result of the detected vertebral masks and contours of the corresponding vertebral bodies. F. The visualization result of the final AI result. The auxiliary parallel lines of superior and inferior endplates are separately displayed in blue color and pink color. The most deviated cephalad end and caudad end vertebrae are separately framed with purple and green boxes.

Under the precondition of the successful test, there were altogether 6099 vertebral bodies detected by the AI program. In further manual review, 48 redundant vertebral bodies and 17 missed vertebral bodies were determined. The sensitivity of AI vertebra detection was 99.74%, and the positive predictive value was 99.28%. In terms of the identification of the vertebral segment, 5340 (87.56%) vertebral bodies were correctly identified, and other 728 vertebral bodies were identified as their contiguous segments. Overall, 6068 (99.49%) results of the segment identification were located within the deviation of one vertebral segment.

The value difference between the Cobb angle obtained by the AI program and that measured manually was 0.4020±0.8703 and the intraclass correlation coefficient was 0.915 with P < 0.001. The Bland-Altman Plot (Figure 3) displayed that the vast majority of the points were located around the average value and between the 95% confidence interval (CI).

Figure 3.

The Bland-Altman Plot of the Cobb angle results measured by the two methods. The x-axis displays the average of the two measures, while the y-axis displays the difference between the Cobb angle obtained by the AI program and that measured manually. The solid red line represents the average value and the light blue dashed lines indicate the 95% confidence interval. Each scattered point represents the result of a test chest X-ray image.

According to the diagnostic criteria of scoliosis, the diagnostic results of the AI method and the manual method were represented in Table 1. Regarding the manual Cobb angle result as the ground truth, this AI scoliosis diagnosis achieved an accuracy of 98.37% in the test set, with a specificity of 98.73% and a sensitivity of 88.24%. The P value of McNemar’s test was 0.289, suggesting there was no significantly difference between the two methods in scoliosis diagnosis within the dataset. In addition, the kappa coefficient of the two diagnostic methods was 0.781. As shown in Figure 4, the area under the ROC curve (AUC) of the AI scoliosis diagnosis was 0.979.

Table 1.

Crosstabulation table of the two methods in scoliosis screening

		Manual measurement		Total
		scoliosis	non-scoliosis
AI measurement	Scoliosis	15	6	21
	non-scoliosis	2	468	470
Total		17	474	491

Figure 4.

The receiver operating characteristic curve of the AI program in screening scoliosis.

Discussion

Typically, the whole thoracic vertebrae and upper lumbar vertebrae can be displayed on chest radiograph. Even though the lumbar curve of scoliosis tends to be underestimated, chest X-ray is regarded as a feasible approach for scoliosis screening [16,17]. In this study, we chose an open-source chest radiography dataset derived from the same type of device. One positive aspect of using the open-source image dataset was to facilitate comparisons of future studies in passing. Although the Shenzhen CXR dataset was originally set up for the diagnosis of pulmonary tuberculosis [18,19], the spinal signals that provided information on the curve of the spine were unaffected by lung disease. The use of this image dataset was intended to testify the effectiveness of the proposed algorithm. Consequently, CXRs from people aged between 10 and 60, the majority of the dataset, were targeted to support the task, while others were excepted due to the small number of same-age samples.

Annotation of images from the training set and validation set was started at the first thoracic vertebra, from top to bottom. But during the automatic test process, for test set images, a few seventh cervical vertebrae and sporadic other cervical vertebrae were inevitably detected due to the structural similarity, and there were occasional missed detections. Fortunately, these redundant and missed detections had little impact on the overall Cobb angle measurement, but could affect the supplementary identification of the vertebral segment. The identification procedure was also processed by the inference module, which was capable of numbering vertebral segment according mainly to the sequence of detected bounding boxes in most cases. But some minor deviations in segments’ identification, accounting for 12.44% of all, were inevitable when vertebrae were mistakenly detected. Briefly, 99.49% of results of vertebrae identification were within the deviation of one vertebral segment.

The main AI gauging program was a combination of the supervised learning module and the inference module. The supervised learning module was built on the basis of Mask-R-CNN [15]. The Mask-R-CNN network structure after the Region Proposal Network [20] was modified and a branch with 26 critical points designed for endplates was inserted into the framework. And the inference module, the core of our gauging program, possessed the ability to determine the suitable endplate fitted lines according to the locations of bounding boxes, masks, critical points, the central curve and the slopes of the adjacent vertebral endplates.

The trained program showed relatively fast detection speed and good generalization ability for images from the same source. After reprocessing by the inference module, the output results could well cover the vast majority of the test set. Nevertheless, due to image quality or unexpected reasons, occasional bounding box errors came with the output. Although we tried to avoid detecting mistaken vertebral bodies by model optimization and parameter adjustment, there were still isolated cases of bounding boxes with the wrong position that could not be eliminated. Or, to be more specific, there were two images with detection failure among the test set.

In general, an increase of 5 or more on two consecutive radiographs was an indication of curve progression [6]. Therefore, the variability of the Cobb angle measurement > 5° would affect the clinical strategy [8,21]. In the comparison of the AI and manual measurements, the sensitivity of AI scoliosis diagnosis was 88.24%, while the specificity was 98.73%, and the accuracy was 98.37%. And the width of the 95% confidence interval was 3.4114 (Figure 3), less than the 5° threshold of change that might influence treatment decisions. And the ICC of 0.915 indicated the strong agreement between the AI and manual Cobb measurements. The result of McNemar’s test revealed that the difference between these two diagnostic methods was not statistically significant. Additionally, the kappa coefficient of 0.781 and the ROC curve (Figure 4) confirmed the consistency of the two methods in scoliosis diagnosis.

In comparison with previous studies [7-11], our method includes the inference module, which was designed to integrate several strategies for the endplate line fitting. Though increasing complexity, this introduction improves the gauging process’s fault tolerance ability and generalization performance. While measuring the Cobb angle, our method can identify the cephalad end and caudad end vertebrae fuzzily and provide a visually intuitive output presentation (Figure 2). Our approach has shown potential for scoliosis screening on CXR, and the likes of this AI method might get a broader application in the future.

In this study, we aimed to provide an automatic AI approach for Cobb angle measurement on CXR. Compared with the manual measurement, we obtained acceptable results with high similarity in the single CXR dataset. However, even if the distribution of scoliosis in the Shenzhen dataset was close to the previous reports [22,23], the primary limitation was that the majority of images were from people without scoliosis. We therefore only paid attention to the curve with the most lateral deviation on chest radiograph in the designing stage. Besides, CXRs with some special conditions, such as severe scoliosis, spinal deformity, spinal degeneration and so on, were not taken into consideration by this study. Our method should be available for regular scoliosis screening in the single-source chest radiograph dataset, but recognition and measurement of images with all these variations were not limited to just expansion of annotation and training [24,25]. Further investigation is needed for this challenging and complicated task.

Conclusion

In this study, an automatic AI method was developed for the measurement of the Cobb angle and the diagnosis of scoliosis, which was also capable of identifying the approximate vertebral segment. The AI program obtained acceptable results as the screening instrument within a single CXR dataset. It seems to be a promising alternative strategy for scoliosis screening on chest radiograph, but some special and extreme conditions were not considered in our design. Further studies are therefore necessary to improve the algorithm and expand the applicable range.

Disclosure of conflict of interest

None.

Abbreviations

AI

Artificial Intelligence

CT

Computed Tomography

MR

Magnetic Resonance

CXR

Chest X-Ray

Mask R-CNN

Mask Region-Convolutional Neural Network

ICC

Intraclass Correlation Coefficient

ROC

Receiver Operating Characteristic

AUC

Area Under The Curve