Virtual Reality visualization for computerized COVID-19 lesion segmentation and interpretation

Abstract

Coronavirus disease (COVID-19) is a severe infectious disease that causes respiratory illness and has had devastating medical and economic consequences globally. Therefore, early, and precise diagnosis is critical to control disease progression and management. Compared to the very popular RT-PCR (reverse-transcription polymerase chain reaction) method, chest CT imaging is a more consistent, sensible, and fast approach for identifying and managing infected COVID-19 patients, specifically in the epidemic area. CT images use computational methods to combine 2D X-ray images and transform them into 3D images. One major drawback of CT scans in diagnosing COVID-19 is creating false-negative effects, especially early infection. This article aims to combine novel CT imaging tools and Virtual Reality (VR) technology and generate an automatize system for accurately screening COVID-19 disease and navigating 3D visualizations of medical scenes. The key benefits of this system are a) it offers stereoscopic depth perception, b) give better insights and comprehension into the overall imaging data, c) it allows doctors to visualize the 3D models, manipulate them, study the inside 3D data, and do several kinds of measurements, and finally d) it has the capacity of real-time interactivity and accurately visualizes dynamic 3D volumetric data. The tool provides novel visualizations for medical practitioners to identify and analyze the change in the shape of COVID-19 infectious. The second objective of this work is to generate, the first time, the CT African patient COVID-19 scan datasets containing 224 patients positive for an infection and 70 regular patients CT-scan images. Computer simulations demonstrate that the proposed method’s effectiveness comparing with state-of-the-art baselines methods. The results have also been evaluated with medical professionals. The developed system could be used for medical education professional training and a telehealth VR platform.

1. Introduction

The number of people killed by the coronavirus pandemic continues to grow, where over (112,461) confirmed cases were reported in Algeria by the end of February 2021. When we refer to the testing categories, the polymerize chain reaction (PCR) tests are the most accurate form available today [1]. However, it takes around 48 h before the results are known. On the other hand, CT scan images are preferred by an experienced doctor to detect the infected lesions in the lungs. CT scan benefits comprise less cost, provide valuable data, and wide availability [2].

Image segmentation has proven a fundamental part of many processing pipelines in the biomedical imaging field [3], [4], [5], [6]. However, when using clinical, low-resolution lung data [7], image segmentation can be challenging. E.g., in acute respiratory syndrome (COVID-19). Many of the existing algorithms were developed to deal with different stages for facing the COVID-19 outbreak, as well as segmentation techniques [8]. These methods can be categorized into two main types: Classical approaches [9], [10] and AI-based techniques [11]. Ulhaq et al in [12], show a broad survey of computer vision methods for screening the COVID-19 lesions. Most recently, methods based on Deep learning have been developed [13]; yet, just a few of them are sufficiently mature to present efficient detection and accurate segmentation of the lesion [14]. Furthermore, these methods require a higher: duration of development, a huge amount of data, and they are computationally expensive [15]. Moreover, there is a distinct lack of labeled medical images of COVID-19 lesions [16]. Some examples of image segmentation and classification methods in COVID-19 applications are summarized in Table 1 along with results obtained of each approach. The abbreviations used in this table are as follow: AUC = Area Under the Curve, SEN = Sensetivity, ACC = Accuracy, Dice = Sørensen–Dice Coefficient, SPE = Specificity, ROC = Receiver Operating Characteristic, PC = Precision-Recall, CT-IM-D = CT-Scan images Data.

Table 1.

Related works on COVID-19 segmentation approache.

Literature	Segmentation approach	CT-IM	Patients	Performance
Zhou et al. [17]	U-Net and Focal Tversky	473	–	Dice = 83.1%, SEN = 86.7%, SPE = 99.3%
Cao et al. [18]	U-Net architecture	–	2	–
Gozes et al. [19]	Deep Learning CT Image	–	157	AUC = 0.99, SPE = 92.2%
Qiu et al. [20]	MiniSeg	100	–	SPE = 97.4%, Dice = 77.2%
Jin et al. [21]	UNet+, CNN	–	723	AUC = 0.99, SEN = 0.97, SPE = 0.92
Fei Shan et al. [22]	VB-Net	249	249	Dice = 91.6%
Xie et al. [23]	Contextual two stage U-Net	204	–	IOU = 0.91, HD95 = 6.44
Shen et al. [24]	Region growing	–	44	R = 0.7679, P < 0.05
Yan et al. [25]	COVID-SegNet	861	–	Dice = 72.60%, SEN = 75.10%.
Jun et al. [26]	UNet++	46,096	106	SPE = 93.5%, ACC = 95.2%
Zheng et al. [27]	Pretrained UNet	–	540	ROC-AUC = 0.959
Xiaowei et al. [28]	VNET	618	106	ACC = 86.7%
Fan et al. [29]	Semi-Inf-Net	100	–	Dice = 73.9%, SEN = 72.5%
Cheng et al. [30]	2D-CNN	970	496	ACC = 94.9%, AUC = 97.9%
Chen et al. [31]	U-Net Res	110	–	Dice = 94.0%, ACC = 89.0%
Ophir et al. [19]	U-net architecture	–	56	AUC = 0.99, SEN = 98.2%, SPE = 92.2%
Lin Li [32]	U-Net	4356	3,322	SEN = 90%, SPE = 96%

In terms of VR visualization, great progress performances were achieved [33]. Particularly, the entertainment industry (gaming) where great progress was achieved in the last decade, and these developments are expected to radically transform healthcare services [34]. Furthermore, VR technology could improve the ergonomic environments of clinicians in such a way as to improve training process, treatment plan, and support clinical decision [35], [36].

VR could be considered as an effective solution for 3D visualization of medical images since it could provide efficient disease analysis and diagnosis regarding the classic approaches. Virtual Reality (VR) gives an opportunity to immerse users in a fully artificial digital medical environment that involves the human anatomy described in 3D models. Today VR systems overcome classical medical imagery problems with novel 3D imagery visualization techniques [37].

In the ongoing pandemic COVID-19, it has been shown that VR-developed techniques help healthcare-related applications [38], [39]. Another advantage of VR is the possibility to offer valuable learnings for medical students and learners for COVID-19 management situations in hospitals and clinics.

Thus, the primary goal will be to develop an advanced VR visualization platform for automated COVID-19 lesion segmentation and infection measurements, analysis, and diagnosis. In summary, our contributions are:

• •An advanced Virtual Reality-based 3D Visualization system for COVID-19 lesion recognition, measurements, and analysis.
• •Novel optimized unsupervised segmentation procedure through a new cost function maximization.
• •Comparison’s study and validation result by the medical staff of the infected patient by different COVID-19 methods.
• •New African COVID-19 patient VR visualization database.

The remainder of the paper is organized as follows. Section 1 reviews the related works related to COVID-19 segmentation. Section 2 outlines the proposed scheme along with a comparison both on objective and subjective perspectives. Meanwhile medical validation is presented in Section 3 and in terms of the VR visualization in Section 4. We conclude by highlighting the outcome achieved (in the last Section 6).

2. Proposed lung segmentation on CT-images

In this section, the proposed COVID-19 lesion segmentation and VR visualization used are presented in detail. CT-scan images considered one of the best sensing approaches since it allows the physicians and radiologists to identify internal structures and see their shape, size, density and texture [40], [41] in a better settings. Our contribution in the context of COVID-19 includes segmentation and VR visualization of COVID-19 lesions so the radiologists will have a clear idea to provide preliminary analysis and interpretation. Fundamentally, digital image processing is an exciting field of research [4], [42], especially in biomedical and medical research [43], [44], [45]. On the other hand, the segmentation process is the approach to detecting and screening images of diseases such as COVID-19 lesions. The image segmentation then is the focal point of the present work, including VR visualization.

Deciding on COVID-19 infected area in the lung is one of the essential for the correct diagnosis. One issue is to identify which regions are infected and which are healthy precisely. Therefore, radiologists are required to view the 3D extent of the lesion to determine the infected area. Fig. 1 presents the process of 3D visualization of COVID-19 from CT-scan images.

Fig. 1.

The general architecture of our advanced VR COVID-19 visualization process.

Firstly, complete DICOM dataset imagery is collected by a CT-scanner. Secondly, the proposed segmentation approach is applied to segment the lesions of giving CT-scan imagery efficiently. A 3D model is generated where the lungs, bronchi, and COVID lesions are separated; this model will be integrated into the VR platform through Unity 3D software. Finally, six radiologists independently evaluate the quality of the platform using a Likert scale (seven-point scale).

Since CT-images illustrate a low-contrast image and contain some noises. The use of the original CT-images without pre-processing affects the accuracy of image segmentation. Therefore, the original image needs to be: i) denoised with informative preservation; and ii) normalized the local contrast. To deal with this issue, the below equation explains the phenomenon of enhancement adopted.

Let X_i,j be an original image, consisting of L^-th luminance levels, i and j represent the pixel location of an image. $X_{i,j}=\left\{x_0,x_1,\cdots ,x_{x-1}|\forall x\subset I\right\}$ and R_i,j be the lung regions, containing two luminance levels, R_i,j = {x ₀ , x ₁ |∀x ⊂ I}

$$Y_{i,j}={\left(\frac{X_{i,j}}{\delta max{X}_{i,j,\delta }\otimes F_{u,v,\delta }}\right)}^{\alpha }.lo{g}^{\beta }\left(\frac{X_{i,j}}{\delta max{X}_{i,j,\delta }\otimes F_{u,v,\delta }}\right)$$ (1)

$$\mathit{For}\left\{X_{i,j}\in \left\{R_{i,j}=x_1\right\}\right\}$$

where F_u,v,δ represents a directional filter. α represents the gamma correction of the linear luminance normalization. u, v and δ represent the location of a 3-D filter. u is position in a row, v is position in a column, and δ is position in a level. ⊗ denotes a 2D convolution operator. β represents the gamma correction of the non-linear luminance normalization. α can improve the details under the bright regions while β is very sensitive and impacts to dark regions. log(.) represents a common logarithm to base 10, log ₁₀(.) = log(.). To avoid all ambiguity, it is best to explicitly specify log (·) when the logarithm to base 10 is intended.

2.1. Threshold based local mean condition

The accuracy of the proposed segmentation algorithm depends on a segmentation condition. In this sub-section, we defined the mean for local-region segmentation. It can be used to exit the iterative segmentation process or control the accuracy of the segmentation, simultaneously. The mean of regions will be repeatedly changed when the size of regions also gradually changed. To generalize the threshold-based local mean method, we combined two threshold-based local mean functions as a single function. It can be applied for wide-range segmentation-based applications. It can be described as:

µ = ω_Kµ_K + ω_Cµ_C (2)

where µ_K and µ_C represent the Kapur’s entropy [46] mean value and a cross-mean [47] value, respectively. The function balance two thresholds by using weights, ω_K and ω_C.

Kapur et al. [46], [47] introduced the single signal-based entropy of segmented classes to generate optimal threshold values. This thresholding technique is extensively improved and illustrates remarkable performance in many image

segmentation problems. First of all, the signal must be separated into two components: a dark component, Y_i,j ≤ x_t, and a bright component, Y_i,j > x_t. The local Kapur’s entropy can be calculated as:

$$K_L\left(x\right)=\frac{p_L\left(x\right)}{{\sum }_{i=1}^x{p}_L\left(i\right)}\log \left(\frac{p_L\left(x\right)}{1+{\sum }_{i=1}^x{p}_L\left(i\right)}\right)$$ (3)

$$K_U\left(x\right)=\frac{p_U\left(x\right)}{{\sum }_{i=x+1}^{x_{L-1}}{p}_U\left(i\right)}\log \left(\frac{p_U\left(x\right)}{1+{\sum }_{i=x+1}^{x_{L-1}}{p}_U\left(i\right)}\right)$$ (4)

where $p_L$ (x) and $p_U$ (x) represent a probability density function of a dark component and a bright component, respectively. $K_L\left(x\right)$ and $K_U\left(x\right)$ denote a local Kapur’s entropy number of a dark component and a bright component, respectively.

Another local mean is a cross-mean thresholding technique. It was first introduced by Trongtirakul et al. [47] in 2020 for segmenting backlit regions and non-backlit regions. The concept of this technique contains the fundamental of enhancement measure by entropy (EME) method. It optimally separates the entire component into several local components by taking the maximum relationship between the local mean of each component. Also, this technique can be applied for a grayscale-based imaging application like CT-scanned, thermal, night-vision applications including multi-spectral, hyper-spectral, and terahertz imaging applications. The cross-mean thresholding can be calculated as:

$$\mu c={\sum }_{n=1}^k\mathrm{a}rgmax{k}_c(x)$$ (5)

$$K_C(\mathrm{X})={-\stackrel{-}{\mu }}_L(X)log\left(\frac{{\stackrel{-}{\mu }}_L(X)}{{\stackrel{-}{\mu }}_U(X)}\right)$$ (6)

Due to the local mean values on a dark component, Y_i,j x_t, and the local mean value on a bright component, Y_i,j > x_t, contain different initialized position. In general, the mean of the bright component is higher than another component. Therefore, it needs to be normalized their whole luminance range. written as:

$$\stackrel{-}{{\mu }_L}\left(X\right)=\frac{{\mu }_L\left(X\right)-min\left\{{\mu }_L\left(X\right)\right\}}{max\left\{{\mu }_L\left(X\right)-min\left\{{\mu }_L\left(X\right)\right\}\right\}}$$ (7)

$$\stackrel{-}{{\mu }_U}\left(X\right)=\frac{{\mu }_U\left(X\right)-min\left\{{\mu }_U\left(X\right)\right\}}{max\left\{{\mu }_U\left(X\right)-min\left\{{\mu }_U\left(X\right)\right\}\right\}}$$ (8)

where ${\mu }_L\left(X\right)$ and ${\mu }_U\left(X\right)$ represent a region-based local mean of a dark component and a bright component, respectively.

To calculate a region-based local mean, let Y_i,j denote the pre-segmented image. X ₀ and X ₁ represent a binary value of Y_i,j. S denotes the size of a local region, R_i,j, and x_L − ₁ represents the maximum luminance level of an image. The illustrative example and the calculation of region-based local mean can be illustrated in Fig. 2 and written as:

Fig. 2.

Illustrative example of the region-based local mean calculation.

$${\mu }_L\left(X\right)=\frac{1}{S·x_{L-1}}{\sum }_{i=1}^M{\sum }_{j=1}^N\left\{Y_{i,j}\right\}=\left\{Y_{i,j}\in \left\{R_{i,j=X_1}\right\},Y_{i,j}{x}_t\right\};$$

$${\mu }_U\left(X\right)=\frac{1}{S·x_{L-1}}{\sum }_{i=1}^M{\sum }_{j=1}^N\left\{Y_{i,j}\right\}=\left\{Y_{i,j}\in \left\{R_{i,j=X_1}\right\},Y_{i,j}{x}_t\right\};$$

where ω_K and ω_C represents the weight of cross-entropy [46] threshold-based local mean number and cross-mean [47] threshold-based local mean number, respectively. argmax_n(•) represents the n − largest search operator to return the elements of entropy functions in which the function numbers are maximized. Y_i,j represents the pre-segmented image. x_t represents a threshold luminance level, starting from t = 2, 3, …, L − 2. M and N represent the size of an image, and x_L−₁ represents the total number of original luminance levels.

After that, we designed the fast segmentation on luminance levels. Most local segmentation need to be defined the criteria to exit an iterative segmentation process. We used the local mean number as aforementioned in the previous sub-section. Another problem for CT-image segmentation is to avoid small, segmented regions. We set this parameter to control the accuracy of the segmentation process. Occasionally, missing significant parts of the desired region to segment, sometimes due to i) the wrong setting in a CT-scanning machine; or ii) the inappropriate processes of the pre-segmentation, for example, unsuitable filters applied for a denoising process, over-brightness or under-exposure for an image enhancement process. The proposed segmentation process contributes the segmentation performance controlled by two conditions: i) threshold-based local mean; and ii) the minimum pixel number in each luminance level. The proposed segmentation function can be described as:

B_i,j = { x₀, M_i,j ≥ µ, card(M_i,j) > cx₁, otherwise (11)

where M_i,j represents a morphologically open image. The morphologically open image function, imopen, is also available in MATLAB. µ represents a threshold-based local mean. Card (•) represents a cardinality operator. c represents the minimum pixel number in each luminance level. x ₀ and x ₁ represent a non-segmented region and a segmented region, e.g., x ₀ = 0 and x ₁ = 1 illustrate that segmented regions become white and other regions are presented by black. For satisfied segmentation results, we set some parameter in the proposed segmentation algorithm as: α = 2.0, β = 2.0, ω_K = 1.0, c = 10.

2.2. Hyper-parameter tuning

The Otsu method is one of the most well-known methods for automatic image thresholding (segmentation and binarization). It is minimizing the inter-class variance between two foreground and background image components information to choose the threshold value. However, the Otsu has a limitation; for instance, if the class variance is larger, it can miss the weak objects in the image see Fig. 3 (b). Also, it performs purely if the input image has low contrast,

Fig. 3.

Parameter influence of the proposed segmentation method a) Input Image; b) Otsu’s Threshold, xt = 105; c) Kapur’s Entropy, xt = 80, d) KL2-Entropy, xt = 122, λ = 0.75, and e) natural response functions.

salt-and-pepper noise corrupted, or uneven illumination. To solve the problem mentioned above, entropy-based segmentation methods are used. Those existing state-of-the-art entropy-based methods also have some limitations, see Fig. 3. They generate a natural response with a slight difference among various luminance levels. Therefore, these methods, including Kapur’s and Tsaills’ approaches, fail to segment in some cases, as shown in Fig. 3(c-d). The proposed KL2-Entropy method enhances the natural response function on the dark image component (x ≤ 0.5), as shown in Fig. 3. The proposed process generates a notable response when the input image is low-contrast like a CT-scanned image. The (λ) parameter is set up into the equation to control the output segmented regions, as shown in Fig. 3(d).

3. Results

This section compares the proposed segmentation method with the COVID-19 lesion segmentation method on new CT-images from a local hospital containing 224 patients in Algeria who tested positive for the coronavirus. These data are provided with a manual labeled ground truth manually labeled by a medical expert. Then, we show the VR visualization and the effect of the COVID-lesion

on the patient lungs. We pick active contour [49], GraphCut [50], ResNet [51], robust threshold [52], [53], Tunable Weka [54] methods to quantitatively compare the results against the proposed in terms of segmentation quality.

3.1. COVID-SVR dataset

Ct-scan data acts as a vital stage in the VR segmentation and diagnosis platform. Presently, there are many COVID-19 public datasets available but few data from African countries are available. We collect a new COVID-19 segmentation and Virtual Reality (COVID-SVR) dataset to fill in this gap. In this subsection, we introduce the data collection, the professional medical labeling, along with significant statistics of our database as shown in Fig. 4 .

Fig. 4.

COVID-SVR data statistics, (a) The ages; (b) severity level of infection in %

To generate high-quality labeling, we first invited two radiologists to mark as many lesions as possible from infected CT-images based on their clinical experience. As can be seen in the second row of Fig. 4 the COVID-SVR data covers different levels of cases.

3.2. Visual evaluation

Fig. 6 summarizes the usefulness of the proposed COVID-19 approach. Regardless of the size or quality of the input CT scan image, by enhancing the lungs tissue following by applying our segmentation, it is proven visually (see Fig. 6(c) (third row)) that the proposed provides accurate extraction of the COVID lesion comparing to state-of-the-art segmentation methods (Fig. 6(d–h) (forth-to-eighth).

Fig. 6.

Visual comparisons of different methods using MosMedData against our segmentation test set (c); (a) original images; (b) ground truths; (d) Active contour; (e) GraphCut; (f) ResNet; (g) Robust Threshold and (h) tunable Weka.

3.3. Statistical evaluation metrics

The metrics used to measure the segmentation performance include the Dice coefficient (SDC), Intersection-Over-Union (Jaccard Index), and Hausdorff metric. The reason for choosing the Hausdorff metrics is: (i) it measures the distance between sets of points, (ii) it commonly used metrics over geometric objects, (iii) it operates as a measure of dissimilarity between binary images, for which it is well suited both for theoretical and for intuitive reasons, (iv) it gives a good idea of the difference in the optical impressions a human would get from two images, and (v) it uses extensively in computer vision, pattern recognition, medicine, and computational chemistry [56]. Mathematically, the Hausdorff (HD) [57] metric is defined as the similarity between two sets A and B.

$$\mathit{HD}(A,B)=max(h(A,B),h(B,A)$$ (12)

The justification for choosing the Jaccard index (J) is: it measures the similarity between sets of patterns by converting each pattern into a single element within the set; it is conceptually and computationally simple, it produces easily interpretable results, and it is an appropriate measure in a wide variety of domain [57]. Mathematically, the Jaccard Index (the similarity between two sets A and B) is defined as the intersection (∩) size divided by the size of the union (∪) of the two sets A and B: where we say J(A, B) = 1 if |A ∪ B| = 0

$$J(A,B)=\frac{\left|A\cup B\right|}{\left|A\right|+\left|B\right|-\left|A\cap B\right|}$$ (13)

Dice Index is a similarity measure related to the Jaccard index [59]. It defined as:

$$SDC\left(A,B\right)=\frac{2\left|A\cap B\right|}{\left|A\right|+\left|B\right|}$$ (14)

Dice Coefficient. The dice similarity coefficient is a measure of positive overlap

between two binary images. It defines as:

DC = 2·TP 2·TP + FP + FN (15)

Where TP is the number of actual positive pixels in the images, FP is the number of false-positive pixels, and FN is the number of false negatives. It is a similarity measure ranging from zero to one–one meaning the two images are identical. The primary benefit of the is that it does not depend on actual negative values. Mean IoU measure: it calculates the ratio of overlapping area between two images with the intersected area subtracted from their union as shown in:

4. meanIoU

A,B = |A ∩ B| |A| + |B| − |A ∩ B| (16)

A perfect segmentation yields an SDC and J score of 1. On the flip side, the Hausdorff metric measures the degree of mismatch between two sets by measuring the distance of the point of A that is farthest from any point of B and vice versa. Small values of H indicate better segmentation, (h) is the distance between point A and the closest point in set B. The resultant segmentation for a set of samples COVID images is shown in Table 2 . From the mean values listed in the aforementioned table, it has been seen that the means of SDC and Jaccard metrics are significantly higher using the proposed, except in terms of SDC where tunable Weka competes (see Table 3 ).

Table 2.

Statistical segmentation quality measurement by using COVID-SVR Data of the proposed against active contour [49], GraphCut [50], ResNet [51], Gated-UNet [60], Dense UNet [61], Robust Threshold [52], [53], and Tunable Weka [54]. The best results are shown in red.

Segmentation methods	Quality measurements
	Dice (SDC) ↑	Jaccard ↑	Hausdorff ↓
Proposed	0.49	0.46	5.62
Active Contour [49]	0.30	0.32	17.77
GraphCut [50]	0.29	0.29	18.64
ResNet [51]	0.33	0.31	27
Robust Threshold [52], [53]	0.28	0.34	15.81
Gated-UNet [60]	0.44	–	–
Dense-UNet [61]	0.41	–	–
Tunable Weka [54]	0.51	0.43	9.68

Table 3.

Statistical segmentation quality measurement by using MosMedData[63] of the proposed against to state-of-the-art segmentation methods. The best results are shown in red.

Methods	Dice (SDC) ↑	Jaccard ↑	Hausdorff $\downarrow$
Proposed	0.74	0.61	1.62
Active Contour [49]	0.32	0.29	3.23
GraphCut [50]	0.36	0.39	3.62
ResNet [51]	0.43	0.41	3
Robust Threshold [52], [53]	0.53	0.44	2.1
Gated-UNet [60]	0.44	–	–
Dense-UNet [61]	0.41	–	–
Tunable Weka [54]	0.54	0.53	3.68
BiSe-Net[62]	0.7	–	–
A-COVID LIRS&M [8]	0.71	0.57	–

Visual assessments for the lung segmentation are given in Fig. 5  and Fig. 6 respectively on the COVID-SVR and MosMedData. These examples illustrate those segmented regions using state-of-the-art methods make different mistakes. Furthermore, some areas are wrongly predicted to be a lesion or healthy tissue by most of the approaches or even all of them. Concerning clinical practice, the near to ground truth segmentation of lesion is the most important that showcases the severity of the disease. Thus, from the qualitative results provided by the proposed (see Fig. 5, Fig. 6) we notice that the propose (Figs. 5 and 6c) provide a near segmentation to ground truth.

Fig. 5.

Visual comparisons of different methods using COVID-SVR Dataset against our segmentation method (c); (a) original images; (b) ground truths; (d) Active contour; (e) GraphCut; (f) ResNet; (g) Robust Threshold and (h) tunable Weka.

Virtual Reality was considered with CT-scan imaging to provide a comprehensive display of lungs with COVID-19 detailed lesions. The results obtained from the segmentation of CT-scan images are used as processed data to generate a 3D COVID-19 lesions visualization system. Although our segmentation method provides a reasonable segmentation, the method proposed is less time-consuming and doesn’t require a high-performance graphical card. This makes

it possible to perform segmentation and 3D visualization faster. Our approach could be suitable for practical situations. Radiologists, in hospitals and clinics, should have the results as quickly as possible in order to deal with a high number of patients, provide adapted treatment, facilitate management of disease and reduce coma situation.

5. VR system design

In this section, we dealt with improving the medical diagnostic proposed method. For that purpose, we provide for radiologists an immersive and interactive VR application to visualize and interact with 3D infected COVID-19 lungs of actual patients. To measure the proposed COVID-based VR application performance, we conducted experiments with doctors and radiologists on a set of patients with COVID-19, including those with mild, moderate, severe, and critical diseases.

Fig. 7 shows the general diagram developed in the Unity game engine, showing the components of the VR developed application. The software part was considered by Unity 3D. In this case, we integrated seven packages: the virtual environment package which integrated optional 3D objects of the scene such as walls, tables, lamps, paintings, etc., three packages for 3D lungs design (3D lung models, 3D bronchus models, and 3D covid-lesion models), interaction package for human–computer interaction management, data manager package for data exchange between packages and 3D scene updates.

Fig. 7.

Packages of the VR application.

The hardware part is composed of Oculus Rift S Head Mounted Display (HMD) (version updated of Oculus Rift [56]) connected to a computer monitor. It integrates sensors to recognize the user’s head movements in the space. Oculus Touch, mouse, and keyboard, manage the 3D interaction between the user and the VR application. The user manipulates can manipulate the 3D infected lungs and go inside the lungs to view the 3D COVID-lesion in more detail (which is possible with 2D CT-scan images). Oculus SDK [64] supports 3D visualization and Hand interaction management.

6. VR application results

We developed a VR application that allows 3D data generation, visualization, and manipulation from medical imaging resources (in our case, we used scanner images). In fact, a DICOM imagery stack was converted to [obj] file and/or [slt]. In our configuration, we used Blender software framework [65] to import [obj] files and generates FBX format [66] directly used to provide 3 lung visualizations.

Fig. 8 shows 3D images obtained using a 3D slicer, with a segmented area of the bronchi, lung with veins, and COVID lesions. The color map that transforms the intensities of the RGB scale into color was manually adjusted so that bronchi, veins, and lesion that contains different views of the object. We used the Unity 3D game engine [67] to import FBX formats and design VR infected lungs where COVID-19 lesions are well-segmented, and 3D described as shown in Fig. 8. Table 4 presents a comparison of the statistics of the progression and growth of the COVID lesion over time of the same person. From the fourth row, we can see that the SDC coefficient is equal to (0.057). This value SDC proves that the lesion’s severity is proportional to the amount of lung tissue destruction, and the lesion grows drastically in 10 days only (as shown in the third row). The lesion’s effect is also higher since the volume (in cc) of the lesion comparison in voxels in CT scans is on the order of (4289.33).

Fig. 8.

VR volume visualization of the automatic segmentation for lung with blood vessels along with COVID-19 lesion.

Table 4.

Comparison and calculation the volume of the lungs and lesion of the effect of COVID-19 on a patient who tested twice by COVID-19

	Segmentation statistics	Number of Voxels
		Lung	Lesion
Day 1		8,010,730	16,077
Day 10		6,621,017	1,306,384
	Comparison
	SDC Coefficient	0.057
	Compare volume (cc)	4289.33

7. Participants

A medical staff composed of six (06) participants (radiologists, doctors, students, and collaborators) volunteered to explore our COVID-19 VR Visualization system and provide their opinions through user experience in a subjective survey. The latter was elaborated with VR experts to measure doctors’ interaction with 3D realistic infected lungs in immersed virtual environments, representing an instant volume-rendered segmented data. In our case, we used Oculus Rift S Head Mounted Display (HMD), to track the user’s head and hands movement. This experimentation could be performed using tablets and smartphones as well.

8. Procedure

The medical staff wears the HMD and visualizes the 3D infected lungs of patients for seven minutes but can be extended if needed. Immersion time was

recorded. Then, no training, nor other explanation, for the VR-naive examiner was performed. After the experience, participants were asked to complete a subjective questionnaire without being disturbed or influenced.

Fig. 9 shows a user using the VR visualization system, which depicts the volume-rendered COVID-19 lesions inside 3D lungs in a virtual room. Furthermore, we provided a VR background surrounding the 3D lung so that participants could be familiar with the VR application.

Fig. 9.

VR visualization of COVID-19 lesions inside 3D lungs.

In the evaluation part, subjects were asked to compare VR infected lung models with scanner images to identify the lesion properties (volume, extension, distribution, location). After the completion of trials, the subjects were asked to answer a short survey about their feelings and fill a questionnaire rated on a 7-point Likert scale.

9. Analysis

Participants were asked to state to the extent to which they agreed or disagreed (a 7-point scale) with 09 statements related to the VR diagnostic COVID-19 platform. The data were also analyzed to explore whether medical staff volunteers who were new to VR experiences had a high comfort score. In this case, it is possible to study the usefulness of the VR application regarding radiologists in their daily work and in extreme conditions. Finally, participants were asked to state what they believed to be the best features. The parameters to evaluate for the experimentation are (parameters from 5 to 9 inspired from [68]). Meanwhile, parameters from 1 to 4 were proposed by authors:

• 1.Immersion is perceived.
• 2.Lesion is well observed.
• 3.Easy to use.
• 4.Could be used for another lesion detection.
• 5.Improve knowledge of disease.
• 6.An enjoyable experience.
• 7.Could help reduce error.
• 8.Enhance understanding.
• 9.Provide a realistic view of clinical case.

10. Discussion

According to question 1, 83% of participants felt Immersed in the virtual world when they firstly manipulate the 3D lung. Among them, one participant was surprised and provided a strongly agreed opinion. More than 80% of experimenters found the 3D COVID-19 lesion clear, visible, and realistic regarding the second question. Simultaneously, all participants found the system easy to use (Question 3). In question 4, 17% of experimenters judged that the application could not be used for another disease detection, where others estimate that the application could be suitable to be applied for other forms of pathology diagnosis. In terms of question 5, 66% believed that 3D technology improves disease knowledge and provides appropriate medical treatment. Similarly, most participants denote those diagnostic errors (Question 6) could be reduced significantly, they also notice that they perceive a realistic 3D view of the COVID-19 lesion and its volume and distribution within the lung (Question 8). Finally, all participants found these experiences enjoyable and improve their understanding of the disease. Moreover, they wish to repeat this experience if they are asked again. Besides, they express their ability to recommend these applications to their colleagues. The summary of the agreement level provided by doctors is illustrated in Fig. 10 .

Fig. 10.

Agreement level provided by medical stuff using 9 statements.

11. Conclusions

The accurate visualization and segmentation of the COVID-19 lesion is a significant stage to help the radiologist in the interpretation of the lesion in a faster way. In this paper, we designed and evaluated an automatic tool for automatic COVID-19 lung infection segmentation and VR visualization using chest CT-scan images. The computer simulations using the MosMedData and COVID-SVR dataset containing (2 2 4) collected from EL-BAYANI center for radiology show better efficiency comparing to the state-of-the-art segmentation approaches, including Active contour, GraphCut, ResNet, Robust threshold, Gated-UNet, Dense-UNet, tunable Weka, BiSe-Net, and A-COVID LIRSM. The proposed algorithm’s performance is measured using the following assessment scores: SDC, Jacquard index, and Hausdorff distance. These contributions prove the prospect of improving diagnosis and visualization for COVID-19 and could support the medical professional to evaluate a possible lesion and abnormalities inside the lungs. After that, the cutting-edge technology using VR is performed to develop a platform of COVID-19 VR viewer, where it was

experimented with by medical staff. As different radiologists with doctors conclude after using our system, the visualization of CT-scan images using Virtual Reality provides a better interpretation of the radiological results and can be a breakthrough for medical treatment planning. According to participants’ opinions, the proposed application provides a more detailed and more realistic view of the COVID-19 lesion’s location, volume evolution, and distribution within the lung. According to a doctor taking part in the trials, the drawback of the proposed approach is that it only gives credible results when the doctor has preliminary information that patients got COVID-19. The doctor said that a lung could have various complex pathologies that differ from the COVID-19 disease. Suppose we provide this application to a doctor without knowing that COVID-19 affected the concerned patient. In that case, he cannot identify the type of pathology even if he uses our VR visualization tool. A medical staff composed of six (06) participants volunteered to explore our COVID-19 VR Visualization system and provide their opinions through a subjective user experience survey. We believe that medical professionals could use this tool to visualize radiological results.

Funding

This research was funded by the national research fund (DGRSDT/MESRS) of Algeria and the Centre de Développement des Technologies Avancées (CDTA).

Author Contributions

S.A. and A.O. conceived the project; methodology, A.O. and T.T; validation, S.B. and A.O, N.Z. formal analysis; investigation, S.A.; data creation, A.O. and M-L.A; Data analysis and preparation, A.O.; writing—review and editing, S.A.; visualization, D.A.; project administration, N.Z.;. All authors have read and agreed to the published version of the manuscript.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.