Automated Classification of Rheumatoid Arthritis, Osteoarthritis, and Normal Hand Radiographs with Deep Learning Methods

Abstract

Rheumatoid arthritis and hand osteoarthritis are two different arthritis that causes pain, function limitation, and permanent joint damage in the hands. Plain hand radiographs are the most commonly used imaging methods for the diagnosis, differential diagnosis, and monitoring of rheumatoid arthritis and osteoarthritis. In this retrospective study, the You Only Look Once (YOLO) algorithm was used to obtain hand images from original radiographs without data loss, and classification was made by applying transfer learning with a pre-trained VGG-16 network. The data augmentation method was applied during training. The results of the study were evaluated with performance metrics such as accuracy, sensitivity, specificity, and precision calculated from the confusion matrix, and AUC (area under the ROC curve) calculated from ROC (receiver operating characteristic) curve. In the classification of rheumatoid arthritis and normal hand radiographs, 90.7%, 92.6%, 88.7%, 89.3%, and 0.97 accuracy, sensitivity, specificity, precision, and AUC results, respectively, and in the classification of osteoarthritis and normal hand radiographs, 90.8%, 91.4%, 90.2%, 91.4%, and 0.96 accuracy, sensitivity, specificity, precision, and AUC results were obtained, respectively. In the classification of rheumatoid arthritis, osteoarthritis, and normal hand radiographs, an 80.6% accuracy result was obtained. In this study, to develop an end-to-end computerized method, the YOLOv4 algorithm was used for object detection, and a pre-trained VGG-16 network was used for the classification of hand radiographs. This computer-aided diagnosis method can assist clinicians in interpreting hand radiographs, especially in rheumatoid arthritis and osteoarthritis.

Introduction

Rheumatoid arthritis (RA) and hand osteoarthritis are two different arthritis that cause pain, function limitation, and permanent joint damage in the hands. Plain hand radiographs are frequently used imaging methods in the differential diagnosis of these two diseases, having different clinical courses and treatments. RA is inflammatory arthritis, which affects about 1% of the population. RA is 2–5 times more common in women than men leading to deformities in the joints and permanent disability [1]. It can occur at any age but is most common between 40 and 70. RA usually onsets insidious, affects all synovial joints, causes pain, stiffness, and swelling especially metacarpophalangeal (MCP) joints, proximal interphalangeal (PIP) joints, and wrist joints. Imaging findings such as periarticular osteoporosis, concentric joint space narrowing, marginal erosions, central erosions, fibrous ankyloses, loss of joint space, bony ankyloses, and deformities are found in patients with RA [1–3]. The treatment of rheumatoid arthritis includes disease-modifying anti-rheumatic drugs (DMARDs) and biologic drugs. Treatment should be started early because erosive lytic changes cannot be relieved by treatment [4].

Osteoarthritis (OA) is a common type of arthritis that affects 10% of males and 13% of females, especially involving the hip, knee, and hand joints. As the populations get older, the incidence of OA rises [5]. Hand OA especially affects the first carpometacarpal joints (first CMC), distal interphalangeal joints (DIP), and PIP. OA causes osteophytes, subchondral sclerosis, and asymmetric joint space narrowing (JSN) in hand joints [6–8]. Treatment of OA includes patient education, lifestyle changes, the use of assistive devices such as splints, the removal of predisposing factors, and, if necessary, a change of profession. Analgesics, topical, and oral non-steroidal anti-inflammatory drugs (NSAIDs) are used in the pharmacological treatment of OA. Early diagnosis improves the patient’s quality of life [9, 10].

Medical imaging is a fundamental method to diagnosis and differential diagnosis of RA and OA diseases. Computer-aided diagnosis (CAD) methods are used to assist physicians in the interpretation of medical images (plain radiographs, magnetic resonance imaging (MRI), digital pathology images). In the CAD method, the physician first makes a routine evaluation of the image, re-evaluates his interpretation with the help of the CAD system, and makes the final decision. Thus, the physician receives a second objective interpretation aid. Some evidence suggests that the inclusion of the CAD system in the diagnostic process provides quantitative support for clinical decisions by reducing inter-observer variations [11–13].

Convolutional neural networks (CNNs) are deep learning architectures and are successful at feature extraction and classification and require less preprocessing. However, big data set is required for CNN training from scratch. If there is not enough data available, data augmentation and transfer learning can be applied [13–16]. Transfer learning is the use of a previously trained model on a new task. It is now a very popular method in the field of deep learning [16]. Nowadays, there are some pre-trained networks used successfully for transfer learning developed using the ImageNet database [16–20]. In this study, a pre-trained VGG-16 network was used for transfer learning. Recently, transfer learning and CNN have been applied in some studies in the medical field [21–23]. Data augmentation is used to increase the amount of data by adding slightly modified copies of existing data and helps to improve model accuracy and prevent model overfitting.

There are some algorithms and methods used for object detection and recognition [24–26]. For deep learning–based object detection, Region-CNN (R-CNN) family, Single Shot Detector (SSD), and YOLO series algorithms are used [27–29]. All these algorithms treat object detection as a regression problem, take a particular image, and simultaneously learn the bounding box (BBox) coordinates and the corresponding probabilities of the class labels. YOLOv4 uses a pre-trained Darknet53 network for object detection. YOLOv4 Darknet is a state-of-the-art object detection system using MS COCO dataset.

YOLO is a powerful and faster algorithm that uses deep learning techniques. YOLO detects and classifies multiple objects from images in real time at 45 frames per second. It sends the entire image to a single CNN. A single CNN predicts multiple bounding boxes with different possibilities on the image. YOLO splits the input image into non-overlapped (S × S) grid cells. Grid cells allow objects to be detected. Each grid cell is responsible for the prediction of B possible bounding boxes and confidence scores [27, 30]. The confidence score is an expression of the presence or absence of any object in the bounding box. If there is no object in the cell, the confidence score will be zero. Otherwise, the confidence score is equal to Intersection over Union (IoU). Each bounding box contains x, y, w, h, and a confidence score. x and y coordinates represent the center point of the bounding box, w and h represent the width and height values. YOLO predicts multiple bounding boxes of different sizes and aspect ratios to capture objects of different shapes and sizes. The non-max suppression algorithm is used to select the best bounding box from multiple projected bounding boxes [31].

This study aims to develop a CAD system to assist physicians in the differentiation of patient hand radiography with rheumatoid arthritis and patient hand radiography with osteoarthritis, and normal hand radiography. The second aim of this study is to develop a fully automated model that reduces preprocessing by using the YOLOv4 algorithm for object detection. Using this CAD model, physicians receive a second objective aid when evaluating hand radiographs. This model can be used by general practitioners as well as experienced physicians such as radiologists and rheumatologists.

Materials and Method

Dataset

Dataset images were obtained from plain hand radiographs of patients examined between 1 January 2012 and 1 March 2021 in the rheumatology outpatient clinic of the Medical Faculty of Kırıkkale University. Conventional hand radiographs (CR) taken in the posteroanterior position have been classified as normal, RA, and hand OA radiographs. Radiographs have been classified by three rheumatologists who were unaware of each other. The radiographs classified as the same by at least two specialists were assigned in that class. (1-Writer, 2-Abdurrahman Tufan, Professor of Medicine, Gazi University, Faculty of Medicine, Department of Rheumatology and 3-Levent Kılıç, Associate Professor of Medicine, Hacettepe University, Faculty of Medicine, Department of Rheumatology).

Object Detection

Radiographs obtained from hospital picture archiving and communication systems (PACS) were in jpg format, and their width and height were different. There were various artifacts (patient name, date, some numbers, directional signs) on plain hand radiographs that could adversely affect training. Therefore, we used the YOLOv4 algorithm to crop the images of both hands from the whole radiograph.

In this study, 50 hand radiographs were trained with YOLOv4 (40 for training, 10 for validation) to obtain the object detector used for preprocessing. These 50 radiographs were labeled by the rheumatologist in the Darknet format. The YOLOv4 config file was configured according to one class, and the pre-trained Darknet53 YOLOv4 weights were used for training. Figure 1 shows the training graph obtained at the end of 2000 iterations. With this object detector and non-max suppression algorithm, all radiographs in the dataset were automatically cropped from the best bounding boxes, and these radiographs were used for the classification tasks. Figure 2 shows the radiography images with bounding boxes.

Fig. 1.

YOLOv4 training graph

Fig. 2.

The bounding boxes obtained with YOLOv4 on the image (left). On the right, the non-max suppression algorithm applied to the same image

Others Class

Apart from RA and OA, some diseases such as psoriatic arthritis, gout, calcium pyrophosphate arthritis, scleroderma, and fractures also cause changes in hand radiographs. The inclusion of an image with an incorrect tag is not conducive to training an automated system. Kim and MacKinnon who worked on hand fractures, proposed to include a third diagnostic category of “inconclusive” to automated systems to cope with uncertainty [32]. In this study, a 4th class has been created as “others.” Distal radius fracture radiographs, foot and knee radiographs, pelvis radiographs, and chest radiographs were added to this class and performed the same preprocessing procedures. Radiographs have randomly split into three: training, validation, and test data (70% as training data, 15% as validation data, and 15% as test data). Table 1 shows the number of training, validation, and test images.

Table 1.

Number of training, validation and test images

	Training	Validation	Test	Total
Rheumatoid arthritis	256	56	56	368
Osteoarthritis	262	57	58	377
Normal	231	51	51	333
Others	242	53	53	348

Transfer Learning, Data Augmentation

When there is not enough data for CNN training from scratch, data augmentation and transfer learning can be applied. Transfer learning is the use of a pre-trained model for a new problem. There are some pre-trained networks with different characteristics trained with natural images in the ImageNet database. In this study, the pre-trained VGG-16 network was used for transfer learning.

Data augmentation is used to increase the amount of data by adding slightly modified copies of existing data and helps to improve model accuracy and prevent model overfitting. In deep learning, methods such as flipping, color-changing, cropping, rotating, adding noise, and random erasing are used to augment the images. In this study, rotation, translation, and flipping were applied to the images for data augmentation.

Statistical Analysis

Accuracy, sensitivity, specificity, precision, and AUC (area under the ROC Curve) results are frequently used as performance metrics in the medical field. These metrics are calculated from the confusion matrix and ROC (receiver operating characteristic) curve. The confusion matrix and ROC curve are obtained during the testing of the trained models. (TP = True Positive; FP = False Positive; TN = True Negative; FN = False Negative).

$$\mathrm{accuracy}=\frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{TP}+\mathrm{TN}+\mathrm{FP}+\mathrm{FN}}$$

$$\mathrm{sensitivity}\left(\mathrm{recall},,,\mathrm{True}\mathrm{Positive}\mathrm{Rate}\right)=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}$$

$$\mathrm{specificity}=\frac{\mathrm{TN}}{\mathrm{TN}+\mathrm{FP}}$$

$$\mathrm{precision}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}$$

Results

This work was carried out on a computer with a GeForce RTX2060 graphics processor. In this study, the Keras TensorFlow environment was used for object detection, and the MATLAB® environment was used for classification. The network performance was measured by the accuracy, sensitivity, specificity, and precision results calculated from the confusion matrix, and AUC calculated from the ROC curve. Figure 3 shows the confusion matrices obtained during testing of the models, and Fig. 4 shows the ROC curves.

Fig. 3.

Confusion matrices obtained during testing of the models: RA rheumatoid arthritis, OA osteoarthritis, N normal

Fig. 4.

ROC curves: RA rheumatoid arthritis, OA osteoarthritis, N normal

Four different dataset groups were created with RA images and normal hand images (group 1), with OA images and normal hand images (group 2), and with RA images and OA images and normal hand images (group 3), and with RA images and OA images and “others” images and normal hand images (group 4) datasets. A pre-trained VGG-16 network was used for training the dataset, and the performance of the network was evaluated with test data not used in either training or validation. Table 2 shows the accuracy, sensitivity, specificity, precision, and AUC test results obtained with the pre-trained VGG-16 model.

Table 2.

Test results obtained with the pre-trained VGG-16 model

		Test results
RA and N	Accuracy (%)	90.7
	Sensitivity (%)	92.6
	Specificity (%)	88.7
	Precision (%)	89.3
	AUC	0.97
OA and N	Accuracy (%)	90.8
	Sensitivity (%)	91.4
	Specificity (%)	90.2
	Precision (%)	91.4
	AUC	0.96
RA and OA and N	Accuracy (%)	80.6
RA and OA and N and other	Accuracy (%)	84.4

RA rheumatoid arthritis, OA osteoarthritis, N normal, AUC area under the ROC curve

Discussion

State-of-the-art applications were used in this study, which was conducted to classify RA, hand OA, and normal hand radiographs. To reduce preprocessing processes, an end-to-end study has been designed using the YOLOv4 algorithm for object detection. During the classification, the performance of the network was improved by applying transfer learning and data augmentation, and by adding the “others” class as the 4th class, the application was provided to be expanded and generalized in the future. The transfer learning method was applied with a pre-trained VGG-16 network for the classification of radiographs, and successful results were obtained.

Some studies have been performed previously for the diagnosis of OA and RA using CNN with plain hand radiographs. We have previously achieved successful results by applying transfer learning with VGG-19, GoogLeNet, and AlexNet networks for OA diagnosis from hand radiographs. The accuracies of the models were 93.2% for AlexNet, 94.3% for GoogLeNet, and 96.6% for VGG-19 [33]. In another previous study, we applied the CNN network architecture consisting of six groups of convolution, batch normalization, rectified linear unit (ReLU), and maximum pooling layers for RA diagnosis from plain hand radiographs. The accuracy of the model was 73.33% [34]. Murakami et al. have achieved successful results in their work for the diagnosis of RA from hand radiographs. The true-positive rate and the false-positive rate of their proposed method were 80.5% and 0.84%, respectively [35]. In this study, we achieved 90.7% accuracy in RA and normal hand radiography classification and 90.8% accuracy in OA and normal hand radiography classification. To the best of our knowledge, this is the first study to classify plain hand radiographs as normal, hand OA, and RA. The addition of the “others” class as the 4th class made the model usable in daily practice and generalizable. In other words, for radiographic evaluation, when radiography is presented to this model other than a normal hand, RA, and OA patient hand radiography, the result will be others. Likewise, if radiography of other sites such as foot radiography, lung radiography is presented to the model, the result will be others. If enough radiography is collected in a disease class such as gout or calcium pyrophosphate arthritis, a new class is created as of 5th class, and so on.

RA and hand OA are two different diseases that cause pain, swelling, tenderness, and loss of function in hand joints. Treatments of both diseases are also different. Plain hand radiographs are frequently used in the diagnosis, differential diagnosis, and monitoring of RA and OA. Plain radiographs are a relatively inexpensive and easily accessible imaging modality. If RA or OA is diagnosed using CR, further imaging methods such as MRI or ultrasonography will not be necessary. MRI is an expensive and time-consuming imaging method, and ultrasonography is an operator-dependent imaging method.

Experienced physicians may not be available in every center where CR can be obtained. Physicians who do not have sufficient experience to evaluate CR can take an active role in the treatment of RA and OA by referring the patient to a specialist using this CAD method. In addition, experienced physicians, such as radiologists and rheumatologists, may also use this CAD method to make their final decisions. Thus, unwanted results that may arise due to work intensity, fatigue, and carelessness, insufficient time may be prevented, the physician receives objective second opinion, and the concern of inter-observer and intra-observer reliability can be reduced.

Weak aspects of this study, our patients had early and late radiological changes due to RA on their radiographs, but we did not calculate the modified Sharp score. A modified Sharp score can be used to determine the damage caused by rheumatoid arthritis in the hand and wrist joints [36]. Likewise, we did not calculate disease activity scores in patients with OA, such as the Kellgren and Lawrence scoring method [37]. If this could be done, it would be seen how successful the model was in the diagnosis of early RA and early OA. Further work is needed in this regard.

We obtained promising results in this study, which we conducted to classify rheumatoid arthritis, hand osteoarthritis, and normal hand radiographs. This CAD method, which we are trying to develop, can be especially helpful to general practitioners who do not have enough experience to evaluate hand radiographs. This study was performed using CRs of patients examined in a single rheumatology center. New studies can be conducted in cooperation with several centers to increase the number of images. If this could be done, the model can be developed and generalized.

Author Contribution

All the authors were fully involved in the preparation of this manuscript and approved the final version.

Declarations

Ethics Approval

Ethical approval certificate was obtained from the Non-interventional Clinical Researches Ethics Board in Kırıkkale University. Certificate date: March 25, 2021, Certificate no: 2021.03.11

Conflict of Interest

The authors declare no competing interests.