Evaluation of 4 three-dimensional representation algorithms in capsule endoscopy images

Abstract

AIM: To evaluate the three-dimensional (3-D) representation performance of 4 publicly available Shape-from-Shading (SfS) algorithms in small-bowel capsule endoscopy (SBCE).

METHODS: SfS techniques recover the shape of objects using the gradual variation of shading. There are 4 publicly available SfS algorithms. To the best of our knowledge, no comparative study with images obtained during clinical SBCE has been performed to date. Three experienced reviewers were asked to evaluate 54 two-dimensional (2-D) images (categories: protrusion/inflammation/vascular) transformed to 3-D by the aforementioned SfS 3-D algorithms. The best algorithm was selected and inter-rater agreement was calculated.

RESULTS: Four publicly available SfS algorithms were compared. Tsai’s SfS algorithm outperformed the rest (selected as best performing in 45/54 SBCE images), followed by Ciuti’s algorithm (best performing in 7/54 images) and Torreão’s (in 1/54 images). In 26/54 images; Tsai’s algorithm was unanimously selected as the best performing 3-D representation SfS software. Tsai’s 3-D algorithm superiority was independent of lesion category (protrusion/inflammatory/vascular; P = 0.678) and/or CE system used to obtain the 2-D images (MiroCam^®/PillCam^®; P = 0.558). Lastly, the inter-observer agreement was good (kappa = 0.55).

CONCLUSION: 3-D representation software offers a plausible alternative for 3-D representation of conventional capsule endoscopy images (until optics technology matures enough to allow hardware enabled-“real” 3-D reconstruction of the gastrointestinal tract).

Core tip: Accurate three-dimensional (3-D) reconstruction of the gastrointestinal tract requires the use of stereo-cameras that can simulate human binocular vision. In the absence of such technology in capsule endoscopy, we rely on software approaches [such as the Shape-from-Shading (SfS) algorithms] to obtain 3-D representation of digestive tract structures. In the present study, we evaluated the use of 4 publically available SfS in capsule endoscopy. 3 experienced/experts reviewers concluded that Tsai’s approach is the best of the four available algorithms.

INTRODUCTION

Capsule endoscopy (CE) has changed our diagnostic approach for small-bowel diseases[1,2]. Although more accurate and of higher diagnostic yield than other modalities[3,4], there are still occasions where pathology is either missed or misinterpreted[5-7]. Furthermore, reports have shown that three-dimensional (3-D) reconstruction can facilitate diagnosis by enhancing textural features of mucosal structures or intestinal abnormalities[8,9]. However, accurate 3-D reconstruction of the gastrointestinal (GI) tract requires the use of stereoscopic cameras that can simulate human binocular vision[10,11]. With the current level of technological investment in CE though i.e., camera size, packaging constraints and power consumption, accurate 3-D imaging of the intestinal lumen in small-bowel capsule endoscopy (SBCE) is still unfeasible[9,12].

Therefore, software approaches that offer 3-D representation of conventional monocular two-dimensional (2-D) CE frames have been developed[13] and proposed for use in CE[14]. Such approaches e.g., Shape-from-Shading (SfS) algorithms, are members of a family of shape recovery algorithms called shape-from-X techniques (Figure 1)[13]. Given a single 2-D image, these algorithms recover the shape of objects using the gradual variation of shading[13]. Essentially, surface ‘‘reconstruction’’ with SfS is achieved through a mathematical representation that is inverted in order to recover dense surface distance and normal information by the gradual variation of shading[13]. We were able to retrieve 4 publicly available SfS algorithms[15-18]. To the best of our knowledge, no comparative study with images obtained during clinical SBCE has been performed to date[19]. We aimed to evaluate the 3-D representation performance of 4 publicly available SfS algorithms by comparing them with their equivalent 2-D images of small-bowel structures/lesions obtained during SBCE, in order to identify the algorithm more helpful in facilitating identification and distinction between lesion and surrounding mucosa.

Figure 1.

Shape-from-Shading function. Capturing a surface using a camera removes depth information. Shape-from-Shading (SfS) techniques try to reproduce the missing depth information from a given two-dimensional (2-D) image.

MATERIALS AND METHODS

Between January 2011 and January 2012, 262 SBCE procedures were performed at the Royal Infirmary of Edinburgh (tertiary referral centre for CE for the southeast of Scotland, United Kingdom) in 249 patients (mean age: 52.6 ± 12.1 years), as already described elsewhere[9]. Out of them, 140 were performed with PillCam^®SB2 (Given^®Imaging Ltd., Yokneam, Israel) and 122 with MiroCam^® (IntroMedic^®Co, Seoul, South Korea). A total of 54 were selected images (27 obtained with MiroCam^® and 27 with PillCam^®SB) on the basis of the overall quality i.e., brightness, absence of air bubbles, debris, or opaque luminal fluid and clarity of findings (lesions or structures). Thereafter, images were classified in the following image groups: (1) vascular lesions i.e., angioectasias (n = 16); (2) inflammatory lesions i.e., ulcers, erosions, aphthae, cobblestone, fold and/or villous oedema (n = 18); and (3) protruding lesions/structures i.e., polyp/mass, nodular lymphoid hyperplasia, cluster of focal lymphangiectasia, chylous cysts, and ampulla of Vater, (n = 20).

3-D image representation software

All selected images were reconstructed in 3-D by means of all 4 SfS algorithms. Three reviewers (Rondonotti E, Mandelli G, Koulaouzidis A) with extensive CE experience and blinded to each other participated in this study. In order to facilitate the evaluation process, a Mathworks^© Matlab program with a graphic user interface (GUI) was developed (Figure 2; a video presenting the evaluation process is provided as supplementary material via this link: https://dl.dropboxusercontent.com/u/7591304/EvaluationVideo.mov). The program consisted of two windows in which the conventional 2-D SBCE image (Figure 2, single frame at the right side/window of the GUI screen) and its corresponding 3-D represented images (four, one for each of the 4 SfS under evaluation) are presented to the reviewer (Figure 2, left side/window of the GUI screen).

Figure 2.

For the evaluation phase, a Mathworks^© Matlab program with a graphic user interface was developed. The program consists of two windows in which the conventional two-dimensional capsule endoscopy image (single frame at the right side/window of the graphic user interface screen) and its corresponding three-dimensional represented images (four, one for each of the 4 shape-from-shading under evaluation) were presented to the reviewer.

The 3-D SfS representations appeared in random order. The reviewers had the ability and freedom to rotate and zoom in each of the 3-D represented images. At the bottom of the GUI screen, a single “task request”: “Choose the 3-D representation you consider most helpful in distinguishing the finding (seen in 2-D) from the surrounding mucosa” appeared. This prompted reviewers to choose one among the four 3-D ‘reconstructed’ images, each generated by a different 3-D algorithm. After selecting the best SfS representation, the reviewer had to click “next” to proceed to the next case. This process was repeated until the program reached the last case after which each separate evaluation was concluded.

Outcome measures

Reviewers were asked to evaluate 54 images. The following subgroup analyses were performed: (1) evaluation of 3-D representation according to the type of finding (vascular vs inflammatory vs protruding); and (2) evaluation according to the system generating the 2-D image (PillCam^® vs Mirocam^®). Furthermore, inter-observer agreement was calculated.

Ethics consideration

This study was conducted in accordance with United Kingdom research ethics guidelines. After review by the local ethics committee further specific ethical review and approval were not required, as the study was considered an evaluation of previously collected endoscopy images, using data already obtained as part of regular clinical care[20].

Statistical analysis

For numerical variables, values are presented as mean ± SD. Where necessary, the Fisher exact test was calculated. A two-tailed P value < 0.05 was considered statistically significant. Inter-observer agreement was calculated using an online kappa calculator (available from http://justusrandolph.net/kappa/) which provides the calculation of Randolph’s free-marginal multirater kappa [21], applicable when raters are not forced to assign a certain number of cases to each category. Values of kappa can range from -1.0 to 1.0, with -1.0 indicating perfect disagreement below chance, 0.0 indicating agreement equal to chance, and 1.0 indicating perfect agreement above chance. More specifically, the inte is classified per kappa as poor < 0.20, fair 0.2-0.40, good 0.41-0.60, very good 0.61-0.80 and, excellent 0.81-1.00[22]. All other statistical analyses were performed using a statistical package, StatsDirect, StatsDirect Ltd, Altrincham, Cheshire, United Kingdom.

RESULTS

Of the 4 SfS algorithms, Tsai’s 3-D algorithm outperformed the rest (selected as best in 45/54 images), followed by Ciuti’s (best performing SfS in 7/54 images) and Torreão’s (in 1/54 images); there was a single image for which each reviewer selected (as best performing) a different 3-D representation algorithm. Of note, not once was Barron’s 3-D algorithm selected as best performing (Table 1, Figure 3).

Table 1.

Results of the Shape-from-Shading method per lesion category

SfS method	Vascular		Inflammatory		Protrusion
	PillCam®	MiroCam®	PillCam®	MiroCam®	PillCam®	MiroCam®
Tsai	7	7	7	6	8	10
Ciuti	1	0	1	0	1	4
Torreão	0	0	1	0	0	0
Barron	0	0	0	0	0	0
None selected	0	1	0	0	0	0

Figure 3.

Assessment results for the 4 Shape-from-Shading algorithms per lesion category.

In 26/54 images, Tsai’s algorithm was unanimously selected as the best performing 3-D representation SfS software. Tsai’s 3-D algorithm superiority was independent of lesion category (protrusion/inflammatory/vascular; P = 0.678) and/or CE system used to obtain the 2-D images (MiroCam^®/PillCam^®; P = 0.558). Lastly, the inter-observer agreement was good (kappa = 0.55).

DISCUSSION

In the present study, we compared the performance of 4 publicly available 3-D ‘‘reconstruction’’ algorithms[15-18] (SfS software) using 54 conventional 2-D CE images. The evaluation criterion was subjective i.e., perceived visualisation improvement (3-D representations offered over the corresponding conventional 2-D images) by 3 experienced CE reviewers. Based on this evaluation, Tsai’s algorithm is the 3-D representation model recommended for use in CE. This outcome directly supports Tsai’s SfS model theoretical advantages: (1) able to produce good results for round surfaces, which are the case for most digestive tract shapes; and (2) it behaves quite well with bright surfaces[13].

Depth information is an important aspect of human vision; it helps human brain to analyse and comprehend the surrounding environment. Images captured with conventional (non-stereoscopic) cameras ‘‘discard’’ the 3^rd dimension (depth) as conventional cameras can only save 2 dimensions (height and width). Therefore depth information is lost; and moreover, most imaging algorithms perform less efficiently.

To date, engineers have not been able to equip capsule endoscopes with stereoscopic cameras for the following reasons: (1) packaging/space limitations; (2) low depth resolution of stereoscopic or time-of-flight cameras[22-24]; and (3) power consumption issues. However, it is almost certain that in the foreseeable future these hardware-related limitations will be overcome[11] and eventually 3-D CE will be a commodity. Nevertheless, until hardware changes are widely implemented, several efforts have been made to convert 2-D images into 3-D images (3-D representation or “reconstruction”) through software and dedicated algorithms. There are software algorithms that offer a fair trade-off between 2-D images and hardware-enabled 3-D images. These algorithms are part of a family of shape recovery algorithms called Shape-from-X techniques[13]. Basically a SfS algorithm recovers the shape of objects, given a single monocular image, using the gradual variation of shading[8,13].

SfS algorithms can be divided into four groups: (1) minimization approaches[16-18]; (2) propagation approaches; (3) local approaches; and (4) linear approaches[15]. It is important to remember that each of the 4 SfS algorithms evaluated herein utilizes a different approach to recover the shape from a conventional 2-D image.

More specifically, Tsai et al[15] described an repetitive update of the depth using a linear approximation of the reflectance function. Ciuti et al[16] used a camera model with perspective projection and a light source close to the surface and away from the optical centre to measure depth. Torreão et al[17] applied a linear-nonlinear biological model that mimics neuronal responses to estimate shape. Finally, Barron et al[18] proposed a unified model for recovering shape, reflectance and optional illumination while using local smoothness, global scarcity or entropy, and the absolute colour of each pixel. Although Tsai’s[14,15] method is very straightforward and to an extent simplistic, it provides satisfying results. Cuiti’s et al[16] algorithm, on the other hand, uses a more advanced model (incorporating a camera model with perspective projection) that makes things in the background appear further back than in Tsai’s model (Figure 4).

Figure 4.

Ciuti’s algorithm (left) and Tsai’s method (right). Although Tsai’s method is very straightforward and to an extent simplistic, it provides satisfying results. Ciuti’s et al[16] algorithm, on the other hand, uses a more advanced model that makes things in the background appear darker than in Tsai’s model.

Since for a given 2-D image, light source and surface shape are not known, these algorithms try to model how the 2-D image was created from the 3-D environment to finally produce an approximation this 3-D depth. The above modelling has a significant impact on the resulting 3-D representation. During SfS process additional constraints need to be applied on the surface shape parameters or the light conditions to find the surface characteristics.

In conclusion, we showed previously that 3-D representation software offers a plausible alternative for 3-D representation of conventional CE images (until optics technology matures enough to allow a hardware enabled-“real” 3-D reconstruction of the GI tract)[9]. In the present study we compared 4 publicly available SfS methods. 3-D reconstruction is attracting interest in capsule endoscopy[8,9,14,25-28], especially as newly developed and/or under development CE become available, with greater potential (due to imager and optics) for 3-D software[20].

COMMENTS

Background

Over the past decade, conventional endoscope technology has advanced with the use of three-dimensional (3-D) cameras offering increased diagnostic and interventional capabilities. Unfortunately, due to hardware limitations, 3-D small-bowel capsule endoscopy (SBCE) is still an open technological challenge. It is aspired that 3-D SBCE will be able to offer similar benefits to conventional 3-D endoscopy. Therefore, information technology engineers suggested the use of software techniques (Shape-from-Shading, SfS) methods that simulate 3-D reconstruction i.e., 3-D representation in SBCE images. To date, various SfS approaches have been proposed; each aims to retrieve depth information from 2-D images (shape recovery) through different mathematical transformations, hence offering different shape approximations.

Research frontiers

The authors aimed to evaluate the 3-D representation performance of 4 publicly available SfS algorithms by comparing them with their equivalent 2-D images of small-bowel structures/lesions obtained during SBCE, in order to identify the algorithm more helpful in facilitating identification and distinction between the lesion and the surrounding mucosa.

Innovations and breakthroughs

This study, in conjunction with further similar work in the field, is useful in the assessing the potential validity of integrating 3-D representation in capsule endoscopy reviewing software.

Applications

Software-enabled 3-D representation is a promising approach that enables 3-D imaging at no additional cost. The authors have shown that SfS application leads to improved visualisation in SBCE and is it likely to be of use in certain clinical scenarios, like the ‘mass or bulge’ question.

Peer review

An interesting paper dealing with software and capsule endoscopy.