Cast suppression in radiographs by generative adversarial networks

Abstract

Injured extremities commonly need to be immobilized by casts to allow proper healing. We propose a method to suppress cast superimpositions in pediatric wrist radiographs based on the cycle generative adversarial network (CycleGAN) model. We retrospectively reviewed unpaired pediatric wrist radiographs (n = 9672) and sampled them into 2 equal groups, with and without cast. The test subset consisted of 718 radiographs with cast. We evaluated different quadratic input sizes (256, 512, and 1024 pixels) for U-Net and ResNet-based CycleGAN architectures in cast suppression, quantitatively and qualitatively. The mean age was 11 ± 3 years in images containing cast (n = 4836), and 11 ± 4 years in castless samples (n = 4836). A total of 5956 X-rays had been done in males and 3716 in females. A U-Net 512 CycleGAN performed best (P ≤ .001). CycleGAN models successfully suppressed casts in pediatric wrist radiographs, allowing the development of a related software tool for radiology image viewers.

HIGHLIGHTS

1. Plaster casts in pediatric wrist radiographs can be successfully suppressed through generative adversarial network (GAN) computer vision algorithms.

2. CycleGAN with U-Net architecture and a square input size of 512 pixels achieved significantly best cast suppression in our sample dataset containing pediatric wrist radiographs.

3. CycleGAN-based cast suppression could be implemented as a tool in radiology software, enabling to dynamically blend between original and cast-suppressed versions.

INTRODUCTION

Injured extremities commonly need to be immobilized by plaster casts or splints.¹ It might not be suitable or possible to remove a cast before acquiring radiographs, especially when assessing alignment or consolidation in the follow-up of fractures. Unfortunately, radiographs with present casts impair bone visibility due to increased thickness and density of the examined region. To date, there are no reports about cast suppression methods in extremity X-rays.

Artificial intelligence opened a variety of possible solutions for problems in the field of image processing, manipulation, and enhancement. Different generative models were successfully implemented in image-to-image translation tasks in computed tomography² and magnetic resonance imaging,^3–5 specifically variational autoencoders⁶ and generative adversarial networks (GAN).⁷ Variational autoencoders are simple to train but produce blurry images that lack details.^8–10 GANs usually generate sharper images but face challenges in training stability and sampling diversity, especially when synthesizing high-resolution images.⁶ CycleGAN is a GAN type able to achieve image-to-image translation in unpaired samples.¹¹ One of the recent uses of CycleGAN in medicine was bone suppression on chest radiograms by Liang et al.¹²

The contributions of this paper are as follows:

• We are the first to tackle and offer a solution for cast suppression in X-ray images.

• We evaluated the CycleGAN method using rigorous qualitative and quantitative analyses. The process of evaluation yielded the optimal CycleGAN model for the given problem.

• We propose a ready-to-use software implementation for cast suppression based on the proposed method.

MATERIALS AND METHODS

The ethics committee of the Medical University of Graz (IRB00002556) approved this retrospective study, waiving the requirement for informed consent. To make the presented work easier to follow, we provide a diagram consisting of 3 major parts in Figure 1 that will be explained in detail in subsequent sections.

Figure 1.

Illustration of our research: subfigure (A) is representing the input data. Subfigure (B) shows the CycleGAN architecture: generator G is removing cast on the images, while generator F is applying cast from the images. Discriminator D_x distinguishes between fake images with cast and original images with cast, while the discriminator D_y is doing the same for images without having a cast applied. Subfigure (C) depicts the applications of the trained generators G and F.

Dataset

The images were sampled from a pediatric wrist digital radiography dataset of the Medical University of Graz, containing 20 330 images acquired between 2008 and 2018. We selected 9672 images free of radio-opaque foreign materials, split into images with and without cast in equal shares (n = 4836 per group). The mean age was 11 ± 3 years in radiographs containing cast and 11 ± 4 years in castless samples. A total of 5956 X-rays had been done in boys (mean age 11 ± 4 years) and 3716 in girls (mean age 10 ± 3 years). Images were processed as 8-bit grayscale portable network graphic files. Preprocessing included padding of nonsquare images with black pixels, followed by re-scaling to 1024 × 1024 pixels with Lanczos interpolation.¹³ The fingers invariably needed to point to the upper image border. We stratified the radiographs into subsets: train subset (n = 7600), validation subset (n = 636), and test subset (n = 1436). The castless half (n = 718) of the test set served as a reference in quantitative analyses, while the other half containing cast (n = 718) was used in testing cast suppression performance. Having an even number of images with and without the cast is beneficial for the CycleGAN training (it makes it easier to build a dataset and helps with the understanding and debugging of the training process). Example images from the training subset are shown in Figure 1A.

CycleGAN

Our dataset did not contain pairs of identical radiographs (with and without cast), as they cannot be reliably obtained due to variations in positioning a wrist and/or the X-ray equipment. Therefore, we favored CycleGAN¹¹ as a qualified method for unpaired data over DualGAN¹⁴ or HarmonicGAN,¹⁵ which are complex and can be difficult to train.¹⁶ Moreover, because we are interested in creating precisely one output image, we favored CycleGAN over the MUNIT method that uses latent space to generate new samples in the target domain.^17,18

CycleGAN is an unsupervised approach for learning the translation of images from the source domain $X$ (in our case, images with cast) to the destination domain $Y$ (images without cast), and vice versa:

• mapping G from X to Y, such that the distributions attained through generator G(X) are impossible to distinguish from the ones in Y by the discriminator D_y, and

• mapping F from Y to X, such that the distributions attained through generator F(Y) are impossible to distinguish from the ones in X by the discriminator D_x.

Therefore, CycleGAN consists of 4 neural networks: 2 generators neural networks ($G$ and $F$) and 2 discriminators neural networks ($D_x$ and $D_y$) depicted in Figure 1B.

Model training

The learning process for generators $G$ and $F$ is to maximize the probability that discriminators $D_x$ and $D_y$ produce errors. In our case, the generator $G$ aims to remove a cast from the input, while the discriminator $D_y$ must distinguish real images without cast from generated fakes. To achieve these goals, CycleGAN is trained to optimize a complex loss function that includes both generators and discriminators, composed of:

• Adversarial loss is calculated for each of the transformations: $X\to Y \mathrm{ }\mathrm{and}\mathrm{ }Y\to X$ domain. It forces generators to produce realistic images in the target domain to fool discriminators.

• Cycle-consistency loss is solving the problem of vanishing fractures (or any other detail), requiring that the image must remain the same when both generators are applied.

  $$\left(Y_d{}_{\mathit{omain}}\approx G(F\left(Y_{\mathit{domain}}\right)), \text{and}\hspace{0.17em}(X_{\mathit{domain}}\approx F(G\left(X_{\mathit{domain}})\right))\right)$$

• Identity loss is forcing $G\mathrm{ }\mathrm{and}\mathrm{ }F$ to keep the input unchanged, or nearly unchanged if it already belongs to the target domain of the generator.¹⁷

The generatormodels were a ResNet-based topology, alike Johnson et al¹⁹ (as in the original CycleGAN paper), and a U-Net-based topology as proposed by Ronneberger et al²⁰ (as in various implementations of CycleGAN in numerous papers). Minor adjustments were necessary to account for the different quadratic input sizes of 256, 512, and 1024 pixels:

• ResNet 256—composed of network base with 2 downsampling layers and 9 residual blocks and head with 2 upsampling layers.

• ResNet 512—composed of the network base, followed by 12 residual blocks from; and the network head.

• ResNet 1024—could not be trained due to computational constraints.

• U-Net 256—the number of downsampling and upsampling layers is 8.

• U-Net 512—the model architecture is similar to U-Net 256, with the number of downsampling and upsampling layers is set to 9.

• U-Net 1024—the model architecture is similar to U-Net 256, but the number of downsampling and upsampling layers is set to 10.

The discriminators were constant for all trained models by means of PatchGANs, classifying overlapping 70 × 70 pixel pieces as real or fake.²¹ These PatchGANs allow to apply different patterns based on the different subregions instead of a whole image.⁷ As for the training hyperparameters, they are similar to the ones used in the original paper:¹¹

• Cycle-consistency loss influence $\lambda =\mathrm{ }10$,

• Adam optimizer using batch size equal to 1,

• Learning rate $\alpha \mathrm{ }=\mathrm{ }0.002$, constant for the first 100 epochs and then linearly decreased to 0 for the consequent 100 epochs.

Model evaluation

Quantitative assessment

Casts influenced histograms across the whole pixel intensity spectrum in terms of intensity count increases in the higher range. We compared similarities of the average reference histogram that was calculated on the castless test subset images with the generated cast-suppressed histograms of every image containing cast in the test subset, by comparing the 2 functions, $H_1$ and $H_2$, using the following metrics:

• Correlation can range from 1 (perfect positive correlation) to −1 (perfect negative correlation), the first representing the best histogram similarity²²:

  $$d_{\rho }\left(H_1,H_2\right)=\frac{{\sum }_I^{}\left(H_1\left(I\right)-H_1{}^{-}\right)\left(H_2\left(I\right)-H_2{}^{-}\right)}{\sqrt{{\sum }_I^{}{\left(H_1\left(I\right)-H_1{}^{-}\right)}^2{\sum }_I^{}{\left(H_2\left(I\right)-H_2{}^{-}\right)}^2}}\mathrm{.}$$
• Histogram intersection is a standard measure for histogram comparison, showing the intersection area between 2 histograms, with larger histogram intersections meaning more similar histograms^23,24:

  $$d_{{\cap }^{ }}\left(H_1,H_2\right)=\sum _I^{}min\left(H_1\left(I\right),H_2\left(I\right)\right)\mathrm{.}$$
• Chi-squared distance represents the deviation of observed and expected frequencies. Smaller Chi-squared distances resemble more equal histograms²⁵:

  $$d_{{\chi }^2}\left(H_1,H_2\right)=\sum _I^{}\frac{{\left(H_1\left(I\right)-H_2\left(I\right)\right)}^2}{H_1\left(I\right)}\mathrm{.}$$
• Hellinger distance is commonly used for similarity quantification of 2 given distributions, defined as²⁶:

  $$d_H\left(H_1,H_2\right)=\sqrt{1-\frac{1}{\sqrt{H_1{}^{-}{H}_2{}^{-}{N}^2}}\sum _I^{}\sqrt{H_1\left(I\right)\cdot H_2\left(I\right)}}\mathrm{.}$$

The peak signal-to-noise ratio metric, commonly used to measure noise removal, is not applicable to cast suppression. Namely, we do not possess pairs of images of the same study, which are necessary for peak signal-to-noise ratio evaluation (one image with cast and one image without a cast of the same patient/case study). It needs to be noted that the average reference histogram that was calculated on the castless test subset of images had a similar mean and variance of the pixels’ intensity as the training subset which contained a much larger number of images. Hence, the average reference histogram used is representative and can be considered as a ground truth. Furthermore, to minimize the information loss, we have tested a second approach where we compared generated cast-suppressed histograms against every original castless image histogram in the test set. This approach required much more effort but produced similar results to the ones presented in the paper.

Qualitative assessment

Three radiologists with 8, 3 (in training), and 9 years of experience in musculoskeletal radiology independently ranked the generated images from best (1st) to worst (5th) subjectively on 20 image subsets. Each subset contained the original reference image and the 5 model-generated images, randomly named, shuffled, and scaled to 512 × 512 pixels. Within each set, rank sums served as surrogate parameter for model performances, calculated by multiplying rank and occurrence numbers, where lower scores correspond to higher-ranked models.

Statistical analysis

We used descriptive statistics and comparisons of means to analyze quantitative and qualitative results, computed in SPSS Statistics Version 21 (IBM Corp., Armonk, NY, USA). Quantitative metrics were compared with paired samples t tests, subjective image quality rankings with nonparametric Wilcoxon signed-rank tests for paired samples. P-values below .05 were considered statistically significant. It is important to emphasize that each cast-suppressed image histogram was evaluated separately against the averaged histogram of the castless images in the test subset, creating an experiment that can be treated with a paired samples test.

RESULTS

Quantitative assessment

U-Net 512 was the best-performing model in quantitative tests with a correlation of 0.998 ± 0.005, a histogram intersection of 222 503 ± 16 322, a Chi-square distance of 59 451 ± 55 793, and a Hellinger distance of 0.147 ± 0.043. The other models yielded mixed results: U-Net 256 was worst in all metrics, apart from intersection. All differences between the models among each other were statistically significant at the P < .001 level in paired samples t tests, except Chi-square distance between ResNet-256 and ResNet-512 (P = .072), and histogram correlation between ResNet-256 and U-Net 1024 (P = .745). Quantitative results and details are presented in Table 1 and Figure 2.

Table 1.

Comparison of the averaged pixel intensity histograms of the castless test set (reference), compared with the averaged cast-suppressed histograms

Parameter		ResNet-256	ResNet-512	U-Net 256	U-Net 512	U-Net 1024
Valid samples	N	718	718	718	718	718
Histogram correlation	Mean ±SD	0.988	0.993	0.963	0.998	0.988
		±0.021	±0.015	±0.062	±0.005	±0.013
Histogram intersection	Mean ±SD	214 902	207 977	209 344	222 503	212 718
		±22.391	±20 316	±26 658	±16 322	±20 778
Chi-square distance	Mean ±SD	122 012	128 185	271 652	59 451	165 814
		±132 038	±111 497	±348 462	±55 793	±161 269
Hellinger distance	Mean ±SD	0.173	0.199	0.198	0.147	0.188
		±0.061	±0.052	±0.080	±0.043	±0.052

Note: The best results are given in bold.

Figure 2.

Bar charts displaying the quantitative metrics: (A) Histogram correlation and (B) intersection, (C) Chi-square distance, and (D) Hellinger distance. 95% error indicators are presented. All differences between the models were statistically significant at the P < .001 level, apart from the 2 cases indicated (n.s.: not significant).

Qualitative assessment

U-Net 512 was significantly better ranked than the other models with 27 points median (ResNet-256 and ResNet-512 P < .001, U-Net 256 and 1024 P = .001), in contrast to ResNet-512 with 92 median rating points as the worst model. U-Net 256 and 1024 were regarded equivalent (P = .732). Radiologists’ rankings including cross significances are summarized in Table 2.

Table 2.

Subjective image quality rankings by the 3 radiologists

Rank sums	ResNet-256	ResNet-512	U-Net 256	U-Net 512	U-Net 1024
Rater 1	74	82	60	27	57
Rater 2	67	95	60	32	46
Rater 3	71	93	45	33	58
Raters (median)	71	92	52	27	53
Significance (P), Wilcoxon signed-rank test
ResNet-256	-	.014*	.005**	<.001***	.015*
ResNet-512	.014*	-	<.001***	<.001***	<.001***
U-Net 256	.005**	<.001***	-	.001**	.732
U-Net 512	<.001***	<.001***	.001**	-	.001**
U-Net 1024	.015*	<.001***	.732	.001**	-

Note: Rank sums are given in the top section (color-coded from better [green] to worse [yellow]), cross significances between the models in nonparametric Wilcoxon signed-rank tests are shown at the bottom.

^*** P < .001

^** P < .01

^* P < .05.

DISCUSSION

We report a method to lessen superimpositions of plaster casts in pediatric wrist radiographs. CycleGAN models were able to suppress casts with satisfactory performance in quantitative and qualitative metrics. The U-Net 512 CycleGAN algorithm performed best, indicating that an actual cast suppression software might be possible in the future.

Our literature research did not identify any other published study on the topic of cast suppression in radiographs, neither in children nor in adults. There might be distinctions in cast suppression performances between children, adults, or elderly patients due to inherit differences in their bone configurations, mineral densities, and variabilities in age-related fracture mechanisms. We believe that robust algorithms could be developed, given sufficient input data, but it might be necessary to train different algorithms for different age groups to achieve solid outputs.

To demonstrate the benefits of cast suppression, we created a demo application as Jupyter notebook, accessible online (https://github.com/fhrzic/CastTool). In Figure 1C, we show only a small portion of this application, which represents the general concept. The application is able to linearly blend between original and inferred cast-suppressed images by means of a slider input. This way, a user retains full control to adjust and optimize his or her subjective level of cast suppression on-the-fly. We envision similar tools to be implemented into a Picture Archiving and Communication System (PACS) software as a future option. U-Net 512 was the overall best-performing CycleGAN model, ranking first in both histogram comparisons, and radiologists’ evaluations. U-Net 256 and 1024 followed in expert ratings but were ranked 3rd and 4th in the histogram evaluations. We consider the experts’ ratings superior to quantitative analyses, because retaining the original image impression is the key requirement in the current application. In this regard, ResNet models produced more synthetic-looking images that were prone to exhibit artifacts (Figure 3). More specifically, ResNet models invariably left areas of smudging or blurring back that were compromising the outputs. While there is a need for optimization and research on the available CycleGAN models, the whole field is evolving rapidly and models are subject to change. GAN improvements should be monitored closely, because novel algorithms might achieve better output images. Figure 3 demonstrates rare cases where models left parts of the cast unremoved in terms of artifacts in the image. Nonetheless, there would still be a cast suppression effect.

Figure 3.

A mosaic representing several input images sampled from the test set and the associated outputs generated by the models under consideration. We have sampled images in a way that we display the full potential of the models, as well as cases where the models have difficulties with cast removal (part of the cast would still be present in the images as an artifact).

The current manuscript dealt with cast suppression, while GANs could also perform image translations in the reverse direction by adding casts to castless radiographs (Figure 1C). At first glance, superimposing cast might not be a rational use. However, the training of certain neural networks might potentially profit from cast-augmentation.

A limitation of the proposed method is the fixed input size, which implies that images with better resolution need down-scaling to match the GAN requirements. As a consequence, variable amounts of information will be lost in high-resolution images based on the chosen model. Another limitation is that the generated images are indeed high-fidelity realistic versions of the expected content—which are not necessarily true to what the radiograph would capture if the image was taken in a wrist without cast. This could be problematic for input images that are substantially different in the content used for training. We believe that this issue can be circumvented to some extent by utilizing more training samples derived from diverse data sources and institutions. It also needs to be mentioned that images pass through the CycleGAN model, which requires performance-capable hardware including a graphics processing unit with at least 8 GB of memory. Otherwise, inference times would be unacceptably long. This manuscript also did not specifically assess the utility of cast suppression on parameters like depiction of alignment or consolidation. Further prospective studies need to be planned assessing the potential benefits of the proposed method.

In conclusion, cast suppression was successfully performed in radiographs for the first time. We employed CycleGANs on a pediatric wrist trauma dataset and achieved promising results with the U-Net architecture. Although further improvements and validation steps are needed, the proposed cast suppression method has great potential to be implemented as a helpful tool in radiology practice.

FUNDING

This work has been supported in part by the Croatian Science Foundation under the project IP-2020-02-3770 and by the University of Rijeka under the project uniri-tehnic-18-15.

AUTHOR CONTRIBUTIONS

FH, ST, and IŠ contributed to the study concept. Data acquisition was performed by FH, IŽ, and ST. Data analyses and visualization were conducted by FH and ST. FH, IŽ, and ST prepared the manuscript draft. FH, IŽ, ST, and IŠ contributed to reviewing and editing of the manuscript. IŠ supervised the project.

DATA AVAILABILITY

A complete anonymized image dataset of all measurements and anonymized image data is stored and available at the author's institution. It is part of a larger yet unpublished image collection.

 

Institution where the work originated from is the Department of Computer Engineering, Faculty of Engineering, University of Rijeka, Vukovarska 58, Rijeka 51000, Croatia.