An Application of Artificial Intelligence to Diagnostic Imaging of Spine Disease: Estimating Spinal Alignment From Moiré Images

Abstract

The use of artificial intelligence (AI) as a tool supporting the diagnosis and treatment of spinal diseases is eagerly anticipated. In the field of diagnostic imaging, the possible application of AI includes diagnostic support for diseases requiring highly specialized expertise, such as trauma in children, scoliosis, symptomatic diseases, and spinal cord tumors. Moiré topography, which describes the 3-dimensional surface of the trunk with band patterns, has been used to screen students for scoliosis, but the interpretation of the band patterns can be ambiguous. Thus, we created a scoliosis screening system that estimates spinal alignment, the Cobb angle, and vertebral rotation from moiré images. In our system, a convolutional neural network (CNN) estimates the positions of 12 thoracic and 5 lumbar vertebrae, 17 spinous processes, and the vertebral rotation angle of each vertebra. We used this information to estimate the Cobb angle. The mean absolute error (MAE) of the estimated vertebral positions was 3.6 pixels (~5.4 mm) per person. T1 and L5 had smaller MAEs than the other levels. The MAE per person between the Cobb angle measured by doctors and the estimated Cobb angle was 3.42°. The MAE was 4.38° in normal spines, 3.13° in spines with a slight deformity, and 2.74° in spines with a mild to severe deformity. The MAE of the angle of vertebral rotation was 2.9°±1.4°, and was smaller when the deformity was milder. The proposed method of estimating the Cobb angle and AVR from moiré images using a CNN is expected to enhance the accuracy of scoliosis screening.

INTRODUCTION

News reports related to artificial intelligence (AI) can be seen almost daily, demonstrating how much attention AI has received among the general public. Previously raised concerns include the possibility that AI could surpass humans in terms of knowledge and intelligence and that its implementation in society could result in the loss of jobs for many medical doctors. The current position, however, is that “AI is a function or ability to support work in clinical settings.” More specifically, AI can be used to streamline clinicians’ workflow, improve efficiency, and reduce errors. Currently, disagreements exist regarding to what extent AI should be utilized in the medical field, as the precision of AI is considered to be insufficient for medical purposes. Meanwhile, AI-related technology is progressing rapidly, and its speed and efficiency are expected to be further enhanced with advances in computer specifications. Therefore, AI should be utilized in the field of medical care.

APPLICATION OF ARTIFICIAL INTELLIGENCE TO THE DIAGNOSIS OF SPINAL DISEASES

The AI application that first comes to mind is diagnostic imaging. AI can support the diagnostic interpretation of X-ray, magnetic resonance (MR) images, and computed tomography (CT) images to prevent signs of disease from being overlooked. More specifically, AI is expected to perform automatic diagnoses of spinal metastatic tumors, vertebral fractures, and pyogenic discitis. AI is also expected to play a role in diagnostic support for rare diseases and diseases requiring highly specialized expertise, such as trauma in children, symptomatic diseases, and spinal cord tumors. Other possible applications in medical care include support for automatic diagnoses based either on information from medical interviews or on the results of medical interviews and imaging studies.

Kim et al. [1] developed a program to distinguish between tuberculous spondylitis and pyogenic spondylitis on MR images. They used data from 80 patients with tuberculous spondylitis and 81 with pyogenic spondylitis whose diagnoses were confirmed by bacterial culture to develop an AI-based image diagnosis program for the differentiation of the 2 conditions. The diagnostic precision of the program was illustrated by an area under the receiver operating characteristic curve (AUC) of 0.802. For comparison, the mean AUC value for differentiation by 3 radiologists was 0.729. These findings show that the diagnostic precision was comparable between AI and radiologists.

Chmelik et al. [2] developed an AI-based program to segment and classify osteolytic and osteogenic metastatic spinal tumors on CT images. The program had better AUCs for both osteolytic and osteogenic lesions (0.80 and 0.78, respectively) than specialists (0.72 and 0.75, respectively). Other reported characteristics of this program include its ability to differentiate tumors using CT images obtained with different devices and to cover the entire spine, from the cervical vertebrae to the sacrum.

Wang et al. [3] developed a program to detect spinal metastases on MR images using AI. Sagittal T2-weighted MR images from 26 cases of metastatic spinal tumors, including 5 lung cancers and 5 thyroid cancers, were used. This program had a true positive rate of 90%, enabling the number of false positives per case to be reduced to 0.207.

Jamaludin et al. [4] from the Genodisc Consortium developed an AI-based grading program for the Pfirrmann classification of intervertebral discs, intervertebral disc narrowing, spondylolisthesis, severity of spinal canal stenosis, presence or absence of end plate irregularity, and bone marrow changes (Modic changes). The intrarater reliability of the program and a radiologist was good: 0.91 for the Pfirrmann classification, 0.89 for intervertebral disc narrowing, 0.79 for spondylolisthesis, 0.61 for the severity of spinal canal stenosis, 0.65 for end plate irregularity, 0.69 for caudal end plate irregularity, 0.86 for cranial bone marrow changes, and 0.83 for caudal bone marrow changes. This program may potentially be useful for clinical research in the future.

We have developed an AI-based diagnosis system for scoliosis screening using moiré images [5,6]. Herein, we summarize 2 papers on this topic: one on the estimation of the Cobb angle [5], and the other dealing with the estimation of vertebral rotation (VR) [6].

SCHOOL SCOLIOSIS SCREENING SYSTEM USING MOIRÉ IMAGES

In Japan, a moiré imaging system is used in schools as a scoliosis screening tool [7]. The surface of a student’s back is captured by a moiré camera and is represented as band patterns, which indicate the depth of the surface, just like contour lines (Fig. 1) [8]. Since scoliosis and rotation of the vertebrae cause the back to have an asymmetric surface, scoliosis is screened by seeking to identify asymmetric band patterns on moiré images, and this screening method is believed to enhance the effectiveness and the sensitivity of scoliosis detection [9]. However, the rules for diagnosing scoliosis based on band patterns are somewhat ambiguous in their interpretation. Thus, the diagnoses are mainly made by experts. Furthermore, the moiré images only provide qualitative results—that is, whether a patient has scoliosis or not—and the exact Cobb angle cannot be deduced from the band patterns.

Fig. 1.

Estimation of the positions of vertebral bodies by a convolutional neural network (CNN). To develop this system, which can estimate the positions of 17 vertebral bodies on moiré images, moiré image–radiograph pairs from the same people were collected for machine learning by a CNN. 2D, 2-dimensional.

Machine learning has been applied to stratify spinal deformities. Ramirez used a support vector machine to determine the severity of spinal deformities based on images obtained by surface topography [10]. However, the system could not estimate the definite value of the Cobb angle. Convolutional neural networks (CNNs) have remarkably advanced in the past few years. A CNN consists of convolution layers that extract obvious features, pooling layers that reduce the feature complexity, and fully connected layers used to classify features. Any feature of an object can be detected using a CNN. Recently, Yang et al. [11] created a system that can detect scoliosis from unclothed back photos using a CNN. Although the accuracy of this system was high, with an AUC of 0.929, it could not estimate the definite value of the Cobb angle.

Since moiré images describe the contours of the back, which reflect deformations of spinal alignment, spinal alignment could theoretically be estimated indirectly from the band patterns on moiré images. Therefore, we sought to develop a system capable of automatically estimating spinal alignment (Cobb angle and VR) from moiré images using a CNN (Fig. 1).

DATASET FOR ESTIMATION OF SPINE ALIGNMENT BY A CONVOLUTIONAL NEURAL NETWORK

To estimate spinal alignment from a moiré image, numerous moiré images with information about the location of vertebral bodies were required as a dataset for machine learning (Fig. 1). In the first scoliosis screening at schools, students were screened using moiré images. Students suspected to have scoliosis were referred to the second scoliosis screening, where radiographs were taken to determine whether they had scoliosis. The material for the dataset was collected from the second screening; thus, 1 moiré image and 1 standing whole-spine radiograph were obtained from each student. One subject who had congenital scoliosis was excluded from the dataset. We finally collected 1,996 pairs of moiré images and standing whole-spine radiographs. The age of the subjects ranged from 10 to 16 years. To avoid overfitting during the training process, rotation (3° clockwise and counterclockwise) and mirroring were used for data augmentation. Therefore, we prepared a final total of 10,788 moiré image-radiograph pairs, which included images with Cobb angles of 0° to 55° as the training dataset, and 198 moiré image-radiograph pairs, including images with Cobb angles of 0° to 45°, as the test dataset. The training dataset consisted of 50% normal spine images (Cobb angle <10°), 30% spine images with a slight deformity (10° <Cobb angle ≤20°), and 20% spine images with a mild to severe deformity (20° <Cobb angle). The test dataset consisted of 33% normal spine images, 33% spine images with a slight deformity, and 34% spine images with a mild to severe deformity.

ESTIMATION OF 17 VERTEBRAL POSITIONS

We manually marked the position of vertebral bodies by denoting the 4 corners of each vertebral body, from the first thoracic vertebra to the fifth lumbar vertebra. To project the radiograph onto the moiré image correctly, 4 features on the back—both sides of the neck and waist—were utilized as landmarks for manual projection. For machine learning, a CNN (Alexnet) was used.

In the results obtained through machine learning, the MAE of the estimated vertebral position was 3.6 pixels (~5.4 mm) per person. T1 and L5 had smaller MAEs than other levels. Since T1 is located at the center of the neck, and L5 is located at the center of the pelvis, predicting these positions might be easier.

MEASUREMENT OF THE COBB ANGLE FROM THE ESTIMATED POSITION OF VERTEBRAL BODIES

The Cobb angle is a parameter indicating the severity of scoliosis that is defined as the angle between the upper and lower ends of the criterion vertebrae, which are the most tilted vertebrae on the concavity of the curve. Since the CNN only predicted the positions of 17 vertebral bodies, a method for measuring the Cobb angle was required. The following measurement method was used (Fig. 2).

Fig. 2.

Measurement of the Cobb angle from the estimated position of vertebral bodies. To measure the Cobb angle, a curve was first fit to the 17 positions of vertebrae using a cubic B-spline, and then the angles were calculated between the 2 lines perpendicular to the curve at the midpoint between the top criterion vertebra and the vertebra above it.

(1) A curve was fit to the 17 positions using a cubic B-spline.

(2) The 2 contact points of 3 lines perpendicular to the curve at 3 sequential vertebrae (upper, middle, and bottom) were calculated.

(3) The middle vertebra was defined as the criterion vertebra where the side of the point of contact changed.

(4) The angles were calculated between the 2 lines perpendicular to the curve at the midpoint between the top criterion vertebra and the vertebra above it, and at the midpoint between the bottom criterion vertebra and the vertebra below it.

We compared the Cobb angles measured by doctors to those measured using the proposed method. The MAE of the Cobb angles was 2.9°, and the standard deviation was 2.54°. The error might have been associated with interobserver or intraobserver variability in the Cobb angle measurement, which has been reported to vary from 3° to 10° [12,13].

ACCURACY OF THE COBB ANGLE ESTIMATION

Finally, we calculated the Cobb angles based on the 17 CNNestimated positions of the vertebral bodies, and compared them with the Cobb angles measured by doctors as ground truth. The MAE per person between the predicted Cobb angle and the ground truth was 3.42° (Table 1). The MAE was 4.38° for normal spines, 3.13° for spines with a slight deformity, and 2.74° for spines with a mild to severe deformity.

Table 1.

Mean absolute error and standard deviation of estimated Cobb angle

Variable	Mean absolute error ± SD
All	3.42° ± 2.64°
Normal (Cobb angle ≤ 10°)	4.38° ± 3.11°
Mild deformity (10° < Cobb angle ≤ 20°)	3.13° ± 2.22°
Severe deformity (20° < Cobb angle)	2.74° ± 2.37°

SD, standard deviation.

ESTIMATION OF VERTEBRAL ROTATION

Structural scoliosis is usually accompanied by VR. In contrast, VR is usually not present in cases of nonstructural scoliosis (conditional scoliosis), which is a reversible condition caused by muscle spasms, differences in leg length, low back pain, spondylolisthesis, lumbar disc herniation, or psychological reasons. Since adolescent idiopathic scoliosis (AIS) is a form of structural scoliosis, in scoliosis screening, estimation of both the Cobb angle and VR may enhance the accuracy and efficacy of detecting patients who need interventions to treat scoliosis. Thus, we created a system capable of automatically estimating VR from moiré images.

DATASET FOR VERTEBRAL ROTATION ESTIMATION BY A CONVOLUTIONAL NEURAL NETWORK

We added information on the spinous process to the same dataset that was used to estimate spinal alignment [5]. We divided the dataset into 200 pieces of data for the test dataset and 1,765 for the training dataset. To avoid overfitting during the training process, the training dataset was augmented to 36,652 by rotation (5° clockwise and counterclockwise) and mirroring.

ESTIMATION OF VERTEBRAL ROTATION

In this study, the angle of VR (AVR) was defined as the angle formed by a vertical line and the line connecting the center of a vertebral body and the tip of the spinous process, in the transverse plane (Fig. 3). We calculated the AVR from the CNN-estimated positions on the moiré images. However, the estimated positions included 2-dimensional (x, y) data in the coronal plane, without information on depth (z-axis), which is necessary to calculate AVR in the transverse plane. Thus, we inferred a general length from CT data obtained from 20 patients (19 females and 1 male) with severe AIS. The mean age of the teenagers was 15 years (range, 11–19 years), their mean weight was 44.2 kg (range, 33–56 kg), and their mean height was 158 cm (range, 151.2–170.7 cm). The data were normalized by dividing by the spinal length, reflecting an assumption that was made to establish a general model of spinal dimensions.

Fig. 3.

Angle of vertebral rotation. The angle of vertebral rotation was defined as the angle formed by a vertical line and the line connecting the center of a vertebral body and the tip of the spinous process, in the transverse plane. The depth (l) was obtained from adolescent idiopathic scoliosis patients who required surgical treatment.

RESULTS OF VERTEBRAL ROTATION ESTIMATION

After machine learning, the output of the CNN was 34 positions for 2 features: the tip of the spinous process and the center of the vertebral body at each level from 200 moiré images entered into the test dataset.

Fig. 4 shows the estimated spinal alignment and VR of each vertebra. The ground truth was the points marked on the moiré images by doctors. The AVRs of both the ground truth and the CNN-estimated results were obtained by the proposed AVR method. The MAE of distance between the ground truth and the estimated results in the moiré images was 3.89±1.98 pixels (6.8 ±3.6 mm). The proposed method for measuring AVRs was used for both the ground truth and the estimation. The MAE of the AVR was 2.9°±1.4° (Table 2). The MAE of the AVR was larger when the deformity was more severe, but its magnitude did not correlate with the average absolute AVR. Our results indicate that moiré images of spines with small deformities yielded more accurate results than moiré images of spines with large deformities. Since the majority of screened AIS cases involve slight to mild scoliosis, the proposed method of measuring AVR on moiré images is expected to enhance the accuracy of screening for AIS.

Fig. 4.

Estimated vertebral rotation. The black line on the center connects the center of gravity of each vertebral body. The white lines on the center connect the center of gravity and the spinous process. The line on the left indicates the angle of vertebral rotation at each vertebra.

Table 2.

Mean absolute error of absolute vertebral rotation angle between the estimation positions and the ground truth

Vertebral rotation angle	Mean absolute error ± SD
All	2.9° ± 1.4°
AVR
< 5°	2.0° ± 1.8°
≥ 5°, < 10°	4.3° ± 2.5°
≥ 10°, < 15°	6.5° ± 3.2°
≥ 15°	12.7° ± 5.9°

SD, standard deviation; AVR, angle of vertebral rotation.

CURRENT LIMITATIONS

In previous reports on diagnostic imaging, the diagnostic precision of AI-based programs has never dramatically exceeded that of actual radiologists. This is attributable to the fact that AI-based programs are developed using training data based on the results of image interpretation by physicians. To achieve higher precision, the key is to use information other than imaging findings, such as bacterial culture results, blood test findings, and pathological findings. The next issue is the evaluation of minor imaging findings. The clinical application of AI would be dramatically enhanced by improvements in diagnostic precision for conditions such as minor metastatic spinal tumors, pyogenic spondylitis, and vertebral fractures, as well as the ability to evaluate the severity of such findings. Additional data (imaging, blood test, and time-series data) would be required to develop such a program.

The next issue is versatility. The quality of X-ray and MR images can vary considerably depending on the performance of the instrument used for image acquisition. Development of programs capable of using images with varying quality or processed images will be necessary to ensure the ability of making accurate diagnoses, regardless of instrument performance or image quality.

Finally, in the hype cycle (a graphical presentation of the maturity, adoption, and social application of specific technologies) updated in 2018 by Gartner, an IT research and advisory firm, AI was positioned toward the end of the “peak of inflated expectations” period, immediately before the “trough of disillusionment” stage. Nonetheless, AI will continue to be an important technology, and the current capabilities of AI will need to be accurately grasped in order to explore its applications in the medical field.