Fast bilateral weighted least square for the detail enhancement of COVID-19 chest X-rays

Abstract

Background

X-ray is an effective measure in the diagnosis of coronavirus disease 2019. However, it suffers from low visibility and poor details. A plausible solution is to decompose the captured images and enhance the details. The bilateral weighted least square model can be an effective tool for this task. However, it is highly computationally expensive.

Method

In this article, we propose an efficient algorithm for the bilateral weighted least square model. We approximate the bilateral weight with the bilateral grid and then incorporate it into the optimization model. This significantly reduces the number of variables in the linear system. Therefore, the model can be efficiently solved. We employ the proposed algorithm to decompose the input X-rays into base and detail layers. The detail layers are then boosted and added back to the input to derive the detail-enhanced results.

Results

The subjective results indicate that our method achieves higher contrast than the best-performing method ( $442.30>410.09$ , $426.40>403.34$ , $564.51>531.38$ ). Furthermore, our method is highly efficient. It takes 0.92  s to process a 720P color image on an Intel i7-6700 CPU. The objective results derive from the chi-square test indicate that subjects hold more positive attitudes toward our detail-enhanced images than the original X-ray images ( $3.53>2.72$ , $3.42>2.61$ , $3.5>2.56$ ).

Conclusion

We have conducted extensive experiments to evaluate the proposed image detail enhancement method. It can be concluded that (1) our method could significantly improve the visibility of the X-ray images. (2) our method is fast and effective, thus facilitating real applications.

Introduction

The coronavirus disease 2019 (COVID-19) is extremely contagious due to the high basic reproduction number (R0).^1–4 During the last three years, it has spread rapidly across the world. The booming development of COVID-19 has imposed a heavy burden on the control of the disease. Even though the impact of the virus is being alleviated, the number of reported reinfections is increasing.^5–7 A timely diagnosis of COVID-19 could prevent the spread of the virus. However, the dominant variants^8,9 of COVID-19 are characterized by a range of clinical signs of asymptomatic infection,^10,11 which complicates the diagnosis.

Chest radiography (X-ray) is an essential measure for human experts to diagnose COVID-19 disease.^12,13 Moreover, researchers have also attempted to investigate deep learning-based approaches for the diagnosis of COVID-19. Comprehensive studies of these methods can be found in existing surveys.^14,15 However, due to the low dynamic range and contrast of the X-ray images, subtle but meaningful details may be challenging to identify for both humans and machines. Therefore, it is critical to enhance the details so that the visibility of the X-ray images can be improved.

Detail enhancement refers to the process to enhance the details of images. It can be leveraged to improve the clarity of X-ray images. The early attempts on this task are based on the histogram equalization and its variants, such as the adaptive histogram equalization, ¹⁶ contrast limited adaptive histogram equalization,^17,18 and normalized contrast limited adaptive histogram equalization. ¹⁹ The Histogram equalization-based methods could promote the image details, but they may also increase the contrast of noise and suppress the contrast of meaningful structures.

Recent methods for detail enhancement mostly rely on image decomposition with edge-aware smoothing filters, such as the bilateral filter,^20,21 anisotropic diffusion filter, ²² guided filter,^23,24 soft clustering filter, ²⁵ $L_1$ reconstruction filter, ²⁶ $L_0$ filter, ²⁷ generalized smoothing filter, ²⁸ and weighted least square filter. ²⁹ The decomposition-based methods consider the pixels in the neighborhood during the enhancement process, thus could alleviate the problems of the histogram equalization-based methods.

It is worth noting that, among the decomposition-based methods, the weighted least square (WLS) filter ²⁹ is shown to be particularly useful for detail manipulation, that is, it can significantly enhance the details without causing the gradient reversal or halo artifacts. However, the weighted design only involves the range, making it less edge-aware. Moreover, similar to many methods based on global optimization, it is highly time-consuming, as it requires to solve a linear system with a large inhomogenous Laplacian coefficient matrix.

With the booming advancement of the deep learning technique, we are seeing significant progress in the field of medical data/image processing, such as COVID-19 prediction, ³⁰ segmentation of cervical cytology, ³¹ thyroid ³² and iris, ³³ pulse extraction, ³⁴ and breast cancer detection. ³⁵ Different from these researches, our objective is to enhance the details of the COVID-19 chest X-ray images, thus aiding the diagnosis. We intend to develop a method that is not learning-based, and can be more stable and interpretable.

To facilitate our objective, we develop a novel decomposition-based image detail enhancement method, where we extract the detail layer, boost it, and add it back to the input image. The main contributions of this article are:

1. We propose a bilateral WLS model for the task of the X-ray image detail enhancement. The bilateral weighted scheme involves not only the range, but also the domain, thus can be more edge-aware.

2. We propose a fast solution to our bilateral WLS model, where we approximate the bilateral weight with the bilateral grid. This significantly reduces the size of the problem, thus can be highly efficient.

3. We have conducted subjective and objective experiments to evaluate our method. The subjective results show that our method achieves better contrast than existing methods. The objective results indicate that our method can improve the visibility of COVID-19 chest X-ray images.

The rest of the article is organized as follows. In the “Literature review” section, we summarize relevant methods. The “Method” section covers the details of the proposed method. Both objective and subjective experiments are presented in the “Experimental settings” section and both objective and subjective evaluations are presented in the “Experimental results” section. We discuss the proposed model in the “Discussion” section. Finally, we conclude the article and present future directions of research in the “Conclusion and future work” section.

Literature review

The most important step in our detail enhancement method is image decomposition. In this section, we will focus on various smoothing filters for image decomposition, which can be mainly divided into two categories, namely local and global methods. The former ones are based on explicitly define filter kernels, while the latter ones are based on sophisticated global optimization models.

Local methods

The bilateral filter ³⁶ is perhaps the most prevalent non-linear filter for edge-preserving smoothing. It consists of a spatial kernel and a range kernel. Both of the two kernels are mostly Gaussian functions that are used to penalize pixels that are distant in space and range. Even though there are various artifacts, such as halos and gradient reversals, the bilateral filter is still effective in practice. The main impediment is the high computational cost.

Over the last decade, researchers have made various attempts to improve the efficiency of the bilateral filter. Most of the methods are built upon range quantization,^37,38 range decomposition,^39,40 or spatial subsampling. ²⁷ These strategies significantly accelerate the bilateral filter, but they may not be able to improve the quality of image decomposition over the naive bilateral filter.

Another popular local method is the guided image filter. ²³ The basic assumption in the work is that there exist local linear relations between the corresponding patches of the output and guidance images. Based on this assumption, the authors build a set of local linear regression problems that have closed-form solutions. The guided image filter can be computed efficiently. However, it suffers from severe halo artifacts in image decomposition.

In order to alleviate the halo artifacts that haunt the guided image filter, various strategies have been proposed, such as the anisotropic guided filter, ²⁴ the weighted guided filter with steering kernel, ⁴¹ and the soft clustering guided filter. ⁴² These strategies all focus on the weighted averaging operation in the final step. These variants of guided filter significantly suppress the halo artifacts. However, the improvement is at the cost of sacrificing efficiency.

Global methods

The WLS filter ²⁹ is the pioneering work for global image decomposition. It seeks to minimize the $L_2$ loss between the input and output regularized by the WLS of the output gradients. The model is shown to be particularly useful for detail enhancement, as it alleviates the halo and gradient reversal artifacts. However, the absence of the domain in the weighted design limits its edge-aware power. Furthermore, it can be time-consuming to solve. Though there exist separable 1D approximations,^43,44 they may introduce severe streak artifacts.

The $L_1$ reconstruction model ²⁶ optimizes the $L_2$ fidelity term regularized by the $L_1$ gradient reconstruction loss. The $L_1$ regularization enforces the sparsity of the output gradients, thus better facilitating edge awareness. The authors propose a novel mapping function to compress the small gradients while preserving large gradients, which further promote the edge awareness. However, the lack of the weighted scheme limits its performance.

The iterative least square filter ⁴⁵ explores a non-convex penalty for regularization. It optimizes the $L_2$ fidelity term regularized by the generalized Charbonnier penalty on the gradients. The non-convex penalty function accommodates the gradient distribution of natural images. Therefore, the filter achieves promising performance on various tasks. Furthermore, it is highly efficient. However, it is inclined to the compartmentalization artifacts in detail enhancement.

The generalized smoothing filter ²⁸ is designed to provide a framework for various edge-aware image processing tasks. The model is based on a novel truncated Huber penalty function, which is friendly to weak edges. Therefore, the resultant filter is strongly edge-aware. However, it tends to sharpen the salient edges, thus may be subject to the gradient reversal artifacts. Moreover, the piece-wise objective function can be computationally expensive to solve.

Different from existing optimization-based image decomposition methods that impose gradient regularization, the soft clustering filter ²⁵ models the task as image clustering similar to super-pixel segmentation. The partition matrix derived from the clustering process is then leveraged to construct the affinity matrix for image decomposition. The method accommodates a variety of applications, yet it may not be able to protect thin structures.

From the above discussion, we see that the local methods are mostly fast, but they suffer from poor quality. Moreover, the global methods suffer from artifacts in detail enhancement except for the WLS filter, which on the other hand, is time-consuming to solve. In this article, we propose a global method for image decomposition based on the bilateral WLS model. We consider not only the range, but also the domain in the weighted design, thus can be more edge-aware. Furthermore, we propose a fast solution to our model based on the bilateral grid. ³⁸

Proposed method

Bilateral WLS model

The bilateral WLS model seeks to minimize the summed squared error regularized by the bilateral weighted $L_2$ norm of the gradients. The objective function of the model is

$$\underset{X}{min}\sum _i(X_i-P_i{)}^2+\frac{\lambda }{2}\sum _i\sum _{j\in \mathrm{\Omega }(i)}{W}_{i,j}(X_i-X_j{)}^2$$ (1)

where $W_{i,j}$ is the bilateral affinity between pixels $i$ and $j$ and it is formally defined as

$$W_{i,j}=e^{-\frac{\Vert j-i{\Vert }_2^2}{2{\sigma }_s^2}-\frac{{\Vert P_j-P_i\Vert }_2^2}{2{\sigma }_r^2}},$$ (2)

where ${\sigma }_s$ and ${\sigma }_r$ are the spatial and range bandwidths, respectively, which are used to control the amount of smoothing. Different from the weighted scheme in the original WLS model ²⁹ that considers only the range, the bilateral weight scheme incorporates both the range and domain, thus is deemed to be more edge-aware.

Rewriting equation (1) into the matrix form, we have

$$\underset{X}{min}(X-P{)}^T(X-P)+\lambda X^T(H-W)X$$ (3)

where $H$ is the degree matrix derived from $W$ . Specifically, $H$ is a diagonal matrix, where the $i$ th diagonal entry is the sum of the entries in the $i$ th row of $W$ . $H-W$ is the inhomogeneous Laplacian matrix. The fidelity term $(X-P{)}^T(X-P)$ ensures that the filtering output $X$ does not deviate too far from the input $P$ . The regularization term $X^T(H-W)X$ enforces that the output $X$ must be smoothed. The parameter $\lambda$ balances the two seemingly contradicting terms.

Taking the derivative of equation (3) to $X$ , and let it be zero, we have

$$(I+\lambda (H-W))X=P$$ (4)

where $I$ is the identity matrix. Equation (4) is computationally expensive to solve, as it is a linear system with a large sparse coefficient matrix.

Efficient solver

Directly solving equation (4) can be time-consuming. Inspired by the bilateral grid, ³⁸ which is essentially a fast approximation of the bilateral filter, we propose a fast solver to the bilateral WLS model. According to the idea of the bilateral grid, the bilateral weight can be approximated as:

$$W\approx S^{T}BS$$ (5)

where $S$ stands for the splat matrix, and it is for hard quantization, where the entries can be either 0 or 1. $S$ is a $n\times k$ matrix, where $n$ is the number of pixels and $k$ is the number of grids. $S_{i,j}=1$ indicates that pixel $i$ belongs to the grid $j$ and $S_{i,j}=0$ indicates otherwise. The range and domain values are quantized into grids. The matrix $S$ can then be derived by checking which grid the domain and range values of each pixel fall in. $B$ is the matrix for the high-dimensional spatial-invariant Gaussian filtering. $S^T$ is the slice matrix, and it is the transpose of $S$ .

Based on equation (5), equation (3) can be approximated as

$$\underset{X}{min}(X-P{)}^T(X-P)+\lambda X^T(\tilde{H}-S^{T}BS)X$$ (6)

where $\tilde{H}$ is the degree matrix derived from $S^{T}BS$ . Specifically, $\tilde{H}$ is a diagonal matrix, where the $i$ th diagonal entry is the sum of the entries in the $i$ th row of $S^{T}BS$ .

We introduce an auxiliary variable $Y$ , and let

$$X=S^{T}Y$$ (7)

Assume the number of pixels and grids are $n$ and $k$ ( $n>>k$ ), $Y$ ( $k\times 1$ column vector) is in fact the grid representation of $X$ ( $n\times 1$ column vector), then equation (6) becomes

$$\underset{Y}{min}(S^{T}Y-P{)}^T(S^{T}Y-P)+\lambda Y^{T}S(\tilde{H}-S^{T}BS)S^{T}Y$$ (8)

To solve the optimization problem shown in equation (8), we take the partial derivative to $Y$ and let it be zero, then we have

$$(S{S}^T+\lambda S(\tilde{H}-S^{T}BS)S^T)Y=SP$$ (9)

where $S{S}^T$ is a $k\times k$ diagonal matrix, where each diagonal entry is the number of pixels in the corresponding grid, thus it can be evaluated efficiently. Note that equation (9) is also a linear system akin to equation (3). However, since $Y$ contains much fewer unknowns than $X$ , it can be much easier to solve, namely

$$Y={(S{S}^T+\lambda S(\tilde{H}-S^{T}BS)S^T)}^{-1}SP$$ (10)

In this article, we leverage the preconditioned conjugate gradient algorithm ⁴⁶ to solve the linear system.

When $Y$ is solved, we can derive $X$ by a simple matrix–vector multiplication $X=S^{T}Y$ according to equation (7), namely

$$X=S^T{(S{S}^T+\lambda S(\tilde{H}-S^{T}BS)S^T)}^{-1}SP$$ (11)

Since the matrix $S$ is for hard quantization, the output may exhibit blocky artifacts. In order to eliminate the artifacts, we further apply the domain transform filter ⁴⁷ on the results derived from the optimization model. The domain transform filter is extremely fast ( $\text{O}(n)$ ), so it has a neglectable influence on the running time.

System model

To achieve detail enhancement, the input X-ray image is firstly decomposed into the base and detail layers with our fast bilateral WLS filter. The detail layer is then boosted ( $4\times$ ) and added back to the input image to derive the detail-enhanced result. The flowchart of our enhancement method is shown in Figure 1.

Figure 1.

The flowchart of the proposed X-ray image detail enhancement method.

Our detail enhancement process can be mainly divided into three steps, which are detailed as follows:

1. Derive the base layer $X$ with the bilateral weighted least filter described in the previous sections.

2. Extract the detail layer $D$ by subtracting the base layer from the input image $P$ , as shown below:

   $$D=P-X$$ (12)
3. The detail layer is then boosted and added back to the input image to derive the detail-enhanced image, that is,

   $$R=P+tD$$ (13)

   where $t$ is a scalar specifying the extent of boost. Throughout this article, we set $t=4$ . Our detail enhancement method can be sketched as Algorithm 1.

Algorithm 1.

X-ray image detail enhancement

Input: image $P$ , parameters $\lambda$ , ${\sigma }_s$ , ${\sigma }_r$ , $t=4$ ;

1: Solve equation (11) for the base layer $X$

2: Derive the detail layer $D$ with equation (12);

3: Derive the enhanced image $R$ with equation (13);

Output: enhanced image $R$ ;

Our detail enhancement method is highly efficient. Assume that the number of pixels and grids are $n$ and $k$ , and the number of iterations of the preconditioned conjugate gradient algorithm ⁴⁶ is $m$ , deriving the base layers by solving equation (11) takes $\text{O}(m{k}^2+nk)$ time. Deriving, boosting, and adding back the detail layer are element-wise operations, so the running time is $\text{O}(n)$ . Therefore, the overall time complexity of our detail enhancement method is $\text{O}(m{k}^2+nk)$ . On the other hand, if we derive the base layer by solving equation (4), it would take $\text{O}(m{n}^2)$ . Considering that in practice $k<<n$ , our method could significantly accelerate the detail enhancement process.

Experimental settings

We have conducted both objective and subjective experiments to evaluate the proposed X-ray image detail enhancement method. In this section, we will detail the experimental settings from these two aspects.

Objective evaluation

Metrics

For objective evaluation, we leverage the contrast measure as the objective metric to evaluate the quality of the detail-enhanced X-ray images, as shown below

$$C=\sum _{\delta }\delta (P_i,P_j{)}^2{E}_{\delta }(P_i,P_j)$$ (14)

where $P_i$ and $P_j$ are the intensities of neighboring pixels. $\delta (P_i,P_j)$ is the absolute difference between $P_i$ and $P_j$ , that is,

$$\delta (P_i,P_j)=|P_i-P_j|$$ (15)

$E_{\delta }(P_i,P_j)$ is the probability distribution of the absolute difference $\delta$ . The contrast measure is a nice indicator of the visibility and the amount of details in an X-ray image.

Furthermore, we record and compare the running time (measured in seconds) of various methods on images with varying resolutions. Note that the running time of all methods is collected on a modern desktop with an Intel i7-6700 CPU, an Nvidia GTX 1070, and 16 GB memory. The running time could be an essential criterion to validate the practicability of the proposed method.

Dataset

The objective evaluation is conducted on the COVID-19 chest X-ray image dataset. ⁴⁸ The dataset is initially developed to advance the detection of COVID-19 using chest X-rays. It contains 317 X-ray images in total, which can be mainly divided into three categories, that is, COVID-19, viral pneumonia, and normal chest X-rays. Our method is not learning-based and it does not require training. We test our method on the 61 images in the test dataset.

Methods

We compare the proposed method to a variety of prevalent methods. The full names, abbreviations, and references of all the methods involved in this experiment can be found in Table 1. We use the official implementations of these methods, which are provided by the authors. For the parameter settings, we best tune the parameters based on the recommended values for all methods.

Table 1.

Brief of the methods in our experiment.

Method	Reference	Abbreviation
Fast bilateral filter	Paris and Durand 38	FBF
Guided image filter	He et al. 23	GIF
Domain transform filter	Gastal and Oliveira 47	DTF
Mutual structure filter	Shen et al. 49	MSF
Weighted least square	Farbman et al. 29	WLS
Anistropic diffusion	Liu and Gong 50	AD
Smoothing sharpening filter	Deng et al. 51	SSF
Iterative least square	Liu et al. 45	ILS
$L_1$ reconstruction filter	Yang et al. 26	L1RF
General smoothing filter	Liu et al. 28	GSF
Soft clustering filter	Liu et al. 25	SCF
Ours	-	Ours

Subjective evaluation

Procedure

For the subjective evaluation, we conduct a user study to validate the effectiveness of the proposed method. We invite a total of 36 doctors to participate in our user study. We randomly choose 20 X-ray images from the COVID-19 chest X-ray image dataset ⁴⁸ and then enhance the detail of the images with our method. The doctors are required to observe and analyze 10 original X-ray images, and the other 10 detail-enhanced X-ray images, through a 32-inches monitor with 2K resolution.

After carefully observing all of the 20 images, the doctors are required to provide responses to a questionnaire, which includes the following questions:

1. I think the images are clear for diagnosis.

2. I think the images contain adequate details.

3. I am satisfied with the presented images.

We adopt a five-point Likert scale ⁵² for the answers to the questions. The doctors need to choose responses from the following: strongly disagree (SD); disagree (D); neutral (N); agree (A); and strongly agree (SA). The ratings for the choices are 1, 2, 3, 4, and 5, respectively.

Grouping information

The 36 doctors are from the departments of respiratory, emergency, and geriatrics. There are 15 males and 21 females. They are randomly divided into two groups, namely the control and the treatment groups. The doctors in the control group are presented with the original X-ray images, and the doctors in the treatment group are presented with the detail-enhanced X-ray images derived from our method. We divide the 20 pairs of X-ray images into two image sets, that is, S1 and S2, with 10 pairs in each set. The control group observing image set S1 is the treatment group observing image set S2. The treatment group observing image set S2 is the control group observing image set S1. In this way, all subjects are exposed to both the original X-ray images and the detail-enhanced image. The detailed arrangement can be found in Table 2.

Table 2.

Grouping information.

Set	Group	Mode	No. of subjects ${}^{\mathrm{a}}$
rowspan="2" align="*"S1	Control ${}^{\mathrm{b}}$	X-ray images	18
	Treatment ${}^{\mathrm{c}}$	Enhanced X-rays	18
rowspan="2" align="*"S2	Control ${}^{\mathrm{c}}$	X-ray images	18
	Treatment ${}^{\mathrm{b}}$	Enhanced X-rays	18

${}^{\mathrm{a}}$ The number of subjects.

${}^{\mathrm{b}}$ The control group observing image set S1 is the treatment group observing image set S2.

${}^{\mathrm{c}}$ The treatment group observing image set S1 is the control group observing image set S2.

Research hypotheses

Through the user study, we want to find out the answers to the following research questions:

1. What were the perceived clarity of the original X-ray images and the detail-enhanced X-ray images?

2. What were the perceived details of the original X-ray images and the detail-enhanced X-ray images?

3. What were the satisfaction levels provided by the original X-ray images and the detail-enhanced X-rays?

The corresponding research hypotheses are as follows:

• H1.The detail-enhanced X-ray images derived from our method are clearer than the original X-ray images.
• H2.The detail-enhanced X-ray images derived from our method contain more details than the original X-ray images.
• H3.The detail-enhanced X-ray images derived from our method improve the satisfaction levels over the original X-ray images.

Data collection and analysis

The doctors’ responses to the questionnaire are used to validate hypotheses H1, H2, and H3 in our user study. We printed the questions on paper so that it would be convenient for collecting the responses. The doctors are required to provide their responses by filling in their answers on the papers.

We record the number of subjects choosing each answer for each of the questions in the questionnaire. We leverage the chi-square test to study hypotheses H1, H2, and H3, as the measurements are discrete variables, and the purpose is to test whether the number of individuals in different categories fits a null hypothesis.

Experimental results

Objective evaluation

Quality

Table 3 shows the contrast measures of the original X-ray images and the enhanced images derived with various methods. The contrast measures are averaged over the entire testing dataset. It can be observed that enhancing the X-ray images with any one of the methods could significantly promote the contrast measures. However, among all the methods involved in this experiment, our method achieves the highest contrast for all the three categories of X-ray images. Therefore, our method could better improve the clarity and details in the X-ray images.

Table 3.

Contrast measures of the detail-enhanced X-ray images derived with various methods.

Method	COVID-19	Normal	Viral pneumonia
Input	75.54	32.66	55.34
FBF	408.97	320.34	496.48
GIF	326.63	275.17	393.82
DTF	384.84	310.56	471.71
MSF	401.63	395.47	479.14
WLS	346.86	274.67	409.78
AD	349.83	287.25	461.92
SSF	407.93	327.70	498.09
ILS	427.23	358.15	484.33
L1RF	389.88	312.62	459.09
GSF	395.00	313.63	475.17
SCF	410.09	403.34	531.38
Ours	442.30	426.40	564.51

FBF: fast bilateral filter; GIF: guided image gilter; DTF: domain transform filter; MSF: mutual structure filter; WLS: weighted least square; AD: anistropic diffusion; SSF: smoothing sharpening filter; ILS: iterative least square; L1RF: $L_1$ reconstruction filter; GSF: general smoothing filter; SCF: soft clustering filter.

The superiority of the proposed method can be also validated from the qualitative comparisons in Figure 2. We observe that enhancing the images with any method could improve the clarity, as any one of the enhanced images has a clearer appearance than the input X-ray image. However, the image derived with our method demonstrates even more details than that of fast bilateral filter (FBF), guided image gilter (GIF), domain transform filter (DTF), and iterative least square (ILS). Furthermore, our filter tends to better preserve the intensity of the original X-ray image compared to WLS, soft clustering filter (SCF), and $L_1$ reconstruction filter (L1RF) (see the black areas).

Figure 2.

Qualitative comparisons on medical image detail enhancement: (a) input, (b) FBF, (c) GIF, (d) DTF, (e) WLS, (f) ILS, (g) L1RF, (h) SCF, and (i) ours. FBF: fast bilateral filter; GIF: guided image gilter; DTF: domain transform filter; WLS: weighted least square; ILS: iterative least square; L1RF: $L_1$ reconstruction filter; SCF: soft clustering filter.

Efficiency

To further validate the practicality of our method, we compare its running time (measured in seconds) to that of other methods. Note that all the recorded time is averaged over 10 runs. As shown in Table 4, our method is highly efficient. It takes 0.92 seconds to process a 720P image, enabling interactive image processing. Through comparison, we can see that our method is faster than most of the methods involved in our experiments. DTF, FBF, GIF, and SCF achieve better efficiency than our method. However, our method delivers much better image quality, as demonstrated in the previous subsection. Finally, our method significantly improves the efficiency of WLS.

Table 4.

Running time of various methods under different resolutions (measured in seconds).

Method	$320\times 240$	$640\times 480$	$1280\times 720$
FBF	0.04	0.25	0.85
GIF	0.05	0.26	0.87
DTF	0.01	0.06	0.15
MSF	0.30	1.34	4.35
WLS	0.45	1.92	6.03
AD	0.92	4.19	15.05
SSF	0.08	0.18	0.42
ILS	0.15	0.55	1.63
L1RF	0.07	0.32	1.08
GSF	2.46	9.65	31.51
SCF	0.06	0.27	0.88
Ours	0.06	0.30	0.92

FBF: fast bilateral filter; GIF: guided image gilter; DTF: domain transform filter; MSF: mutual structure filter; WLS: weighted least square; AD: anistropic diffusion; SSF: smoothing sharpening filter; ILS: iterative least square; L1RF: $L_1$ reconstruction filter; GSF: general smoothing filter; SCF: soft clustering filter.

Subjective evaluation

Image clarity

Figure 3 demonstrates the subjects’ responses to the question “I think the images are clear for diagnosis.” Their ratings from 1 to 5 are used to evaluate the clarity of the images. The results are analyzed with the chi-square test as shown in Table 5. We observe that the subjects in the treatment group (presented with the detail-enhanced images) hold a more positive attitude ( $3.53>2.72$ ) toward the clarity of the images than that of the control group (presented with the original X-ray images). Furthermore, we observe that the difference between the perceived image clarity of the two groups is significant ( ${\chi }^2(4)=9.81$ , $P=0.044<0.05$ , $P$ is the probability of the difference being insignificant). The above findings support hypothesis H1, that is, our detail-enhanced X-ray images show better clarity than the original X-ray images.

Figure 3.

Subjects’ responses to the question “I think the images are clear for diagnosis.”

Table 5.

Chi-square test result for the perceived image clarity (degree of freedom = 4).

	Control	Treatment	Expect	${\chi }^2(4)$
SD (1)	8	2	5	3.6
D (2)	7	5	6	0.33
N (3)	12	8	10	0.8
A (4)	5	14	9.5	4.26
SA (5)	4	7	5.5	0.82
Total	36	36	36	9.81
Mean rating	2.72	3.53

SD: strongly disagree; D: disagree; N: neutral; A: agree; SA: strongly agree.

Image details

Figure 4 illustrates the subjects’ responses to the question “I think the images contain adequate details.” Their ratings from 1 to 5 are used to evaluate the details of the images. The results are analyzed with the chi-square test as shown in Table 6. It can be observed that the average ratings about the details of the image in the treatment group (presented with the detail-enhanced images) are higher ( $3.42>2.61$ ) than that of the control group (presented with the original X-ray images). Moreover, we observe that the difference between the perceived image details of the two groups is significant ( ${\chi }^2(4)=10.78$ , $P=0.029<0.05$ ). The above findings support hypothesis H2, that is, our detail-enhanced X-ray images contain more adequate details than the original X-ray images.

Figure 4.

Subjects’ responses to the question “I think the images contain adequate details.”

Table 6.

Chi-square test result for the perceived image details (degree of freedom = 4).

	Control	Treatment	Expect	${\chi }^2(4)$
SD (1)	9	4	6.5	1.92
D (2)	7	5	6	0.33
N (3)	13	7	10	1.8
A (4)	3	12	7.5	5.4
SA (5)	4	8	6	1.33
Total	36	36	36	10.78
Mean rating	2.61	3.42

SD: strongly disagree; D: disagree; N: neutral; A: agree; SA: strongly agree.

Satisfaction level

The subjects’ responses to the question “I am satisfied with the presented images” are shown in Figure 5. The subjects’ ratings from 1 to 5 are used to evaluate the satisfaction level provided by the images. The results are studied with the chi-square test as demonstrated in Table 7. It can be observed that the satisfaction levels of the subjects in the treatment group (presented with the detail-enhanced images) are higher ( $3.5>2.56$ ) than that of the control group (presented with the original X-ray images). Furthermore, we observe that the difference between the satisfaction levels of the two groups is significant ( ${\chi }^2(4)=10.69$ , $P=0.03<0.05$ ). The above findings support hypothesis H3, that is, the satisfaction levels provided with our detail-enhanced X-ray images are better than that provided with the original X-ray images.

Figure 5.

Subjects’ responses to the question “I am satisfied with the presented images.”

Table 7.

Chi-square test result for the perceived satisfaction levels (degree of freedom = 4).

	Control	Treatment	Expect	${\chi }^2(4)$
SD (1)	8	3	5.5	2.27
D (2)	11	4	7.5	3.27
N (3)	9	9	9	0
A (4)	5	12	8	2.88
SA (5)	3	8	5.5	2.27
Total	36	36	36	10.69
Mean rating	2.56	3.5

SD: strongly disagree; D: disagree; N: neutral; A: agree; SA: strongly agree.

Discussion

In this article, we propose a novel and efficient method to enhance the details of images and explore the COVID-19 chest X-ray image dataset ⁴⁸ as an example to demonstrate the effectiveness of the proposed method. In fact, our method essentially defines a general framework for image detail enhancement, that is, besides the X-ray images, our method can be potentially applied to an even broader range of images, such as computed tomography images, magnetic resonance imaging images, or even natural images. More studies need to be conducted to evaluate our method in enhancing other types of images. Our method is implemented on the CPU, which processes 720P images at interactive rates. It would be beneficial to implement the proposed method on GPU so that it can achieve real-time enhancement for 720P images.

Due to the lack of ground truth labels, we leverage the contrast measure for objective evaluation of the proposed image detail enhancement method. The contrast measure is a good indicator of the details and clarity. However, it does not reflect artifacts or degradation generated in the enhancing process. More efforts need to be spent to build a dataset with ground truth labels for X-ray image detail enhancement. The ground truth labels can be obtained with more extensive user studies, where human experts are invited to select the best-looking images from a set of detail-enhanced images derived from a variety of methods. In this way, one can study the quality of the detail-enhanced images in a more objective sense, using measures such as peak noise-to-signal ratio and structure similarity.

The proposed method can be used to enhance the details of the X-ray images. It can be beneficial to down-streaming diagnosis tasks conducted by either machines or human experts. However, the tones of images are not involved in this article. The X-ray images suffer from low dynamic range. Therefore, the tone can be also important to the visibility of the X-ray images. More studies need to be done to consider the tones of the X-ray images in order to further improve the clarity. Possible directions can be the HDR tone mapping operator ²⁰ and the tone manipulation method. ⁵³ It would be interesting to investigate the proposed fast WLS filter in the tasks of tone mapping and manipulation to see how it can be applied to further improve the visibility of the X-ray images.

The proposed solution to the optimization model shown in equation (11) is highly efficient. However, it is based on the approximation with the bilateral grid, which is essentially a hard quantization technique. Therefore, the results of our filter may exhibit blocky artifacts. Even though the artifacts can be eliminated by post-processing with the domain transform filter, degradation in the quality of the results may be introduced. It would be also interesting to investigate soft quantization or clustering techniques to approximate or define the affinity matrix in the weighted scheme, so that the results can be free of blocky artifacts without post-processing. Possible ways to achieve this objective can be the approximation of the bilateral affinity with spatial subsampling ²¹ or the soft clustering affinity. ²⁵

Conclusion and future work

X-ray images can be essential to the diagnosis of COVID-19, which has brought significant impacts on our lives. However, the original X-ray images may suffer from low dynamic range and low contrast, which would hinder the diagnosis by either human experts or machine learning methods. Aiming at improving the clarity of the X-ray images, we propose an efficient method to enhance the details of the X-ray images. The proposed enhanced method is based on image decomposition with the bilateral WLS model. In order to efficiently solve the model, we propose to approximate the bilateral weight matrix with the bilateral grid so that the large sparse linear system would be simplified into a linear system with much fewer unknowns.

We have conducted extensive experiments to evaluate the proposed X-ray image detail enhancement method. Both objective and subjective evaluations are leveraged in our experiment. For the objective evaluation, we analyze the contrasts of the enhanced images derived with different methods. It is shown that our method could achieve higher contrast than the methods being compared. Furthermore, we compare the running time of different methods in deriving the enhanced images. Our method can process 720P 3-channel images at interactive rates on an i7-6700 CPU. For the subjective evaluation, we conduct a user study that involves 36 doctors. The chi-square test results indicate that the ratings of our detail-enhanced X-ray images are significantly higher than that of the original X-ray images. Therefore, we draw that our method could significantly improve the visibility of the X-ray images. Moreover, it can be applied to real practice, as it is efficient and effective.

Based on the discussion in the previous section, we summarize the future directions of research as follows:

1. We only apply our detail enhancement method in X-ray images, more studies need to be conducted to evaluate our method on a broader range of images.

2. In order for a more objective evaluation, more efforts need to be spent to build a dataset with ground truth labels for X-ray image detail enhancement.

3. The proposed method only deals with the details, more studies need to be carried out to investigate the tone manipulation techniques^20,53 to further improve the clarity.

4. Our fast solution is based on a hard quantization technique, that is, the bilateral grid, more studies need to be conducted to explore affinity designs with soft quantization ²¹ or clustering. ²⁵