Tensor-based Feature Extraction for Pupil Recognition in Cataract Surgery

Abstract

Cataract surgery remains the definitive treatment for cataracts, which are a major cause of preventable blindness worldwide. Adequate and stable dilation of the pupil are necessary for the successful performance of cataract surgery. Pupillary instability is a known risk factor for cataract surgery complications, and the accurate segmentation of the pupil from surgical video streams can enable the analysis of intraoperative pupil changes in cataract surgery. However, pupil segmentation performance can suffer due to variations in surgical illumination, obscuration of the pupil with surgical instruments, and hydration of the lens material intraoperatively. To overcome these challenges, we present a novel method called tensor-based pupil feature extraction (TPFE) to improve the accuracy of pupil recognition systems. We analyzed the efficacy of this approach with experiments performed on a dataset of 4,560 intraoperative annotated images from 190 cataract surgeries in human patients. Our results indicate that TPFE can identify features relevant to pupil segmentation and that pupil segmentation with state-of-the-art deep learning models can be significantly improved with the TPFE method.

I. INTRODUCTION

Cataract surgery addresses the pathologic clouding of the natural lens of the eye through removal of the clouded lens material and replacement with an artificial intraocular lens (IOL) implant. It is one of the most commonly performed surgeries worldwide and essential to addressing preventable blindness. There were more than 20 million cataract surgeries performed in the world in 2015, of which 3.6 million cases were in the United States of America and more than 4.2 million cases were in the European Union [1]. Cataracts most commonly occur as a natural part of the aging process, but can also occur due to trauma, medication side effects, and metabolic disorders. Despite improvements in cataract surgery technology and the phacoemulsification procedure, cataracts are one of the leading causes of blindness worldwide.

Cataracts are accessed surgically by passing instruments through the pupil, which is the natural aperture in the iris tissue. The pupil is pharmacologically dilated at the time of surgery, but pupillary instability during surgery is a major risk factor for cataract surgery complications. As such, pupil recognition in cataract surgery is an essential task that allows surgeons to understand changes in pupil morphology during surgery. However, recognition performance is affected by many factors occurring during cataract surgery, such as variations in illumination, obscuration by surgical instruments, and hydration of cataractous lens material altering the appearance of the pupil itself. In the recent past, with the rapid development of deep learning, studies [2] – [5] proposed methods for pupil recognition in various scenarios, predominantly outside of the context of cataract surgery.

Lee et al. [2] applied UNet model to recognize and separate pupil from nystagmography video achieve accurate trajectory of pupil for diagnosis of various vestibular disorders. The group showed the potential for applying deep learning methods to diagnosing vestibular disorders since the UNet model achieved 94.85% dice coefficient. In other scenarios, Han et al. [3] proposed an indirect use of the CNN for pupil center detection. Accordingly, the pupil region is first segmented by CNN models and the center of the segmented region is then determined. The authors demonstrated that the proposed method outperforms other conventional methods that directly find the pupil center as a regression result. Gowroju et al. [4] presented a modified UNet model to improve the accuracy of pupil segmentation in non-surgical images. The proposed model achieved a notable accuracy when compared to the original UNet model while consuming less time.

In the field of cataract surgery, Sokolova et al. [5] trained and tested a Mask R-CNN on a small (82-image) dataset of cataract surgery frames and showed moderate performance in the pupil segmentation task. This study was limited by the size of its dataset and the need for analysis of performance across varying phases of surgery and in the context of pupil obscuration by a variety of surgical instruments.

In this paper, we propose a novel method called tensor-based pupil feature extraction (TPFE) for pupil recognition in cataract surgery using deep learning models. The proposed method is designed to extract features of the pupil region effectively with the goal of utilizing TPFE’s feature-rich output as input to pupil recognition systems. In order to study its effectiveness, we undertook experiments to evaluate the segmentation performance of a set of deep learning models with and without TPFE pre-processing. In the subsequent sections, we describe the TPFE method, report its impact on pupil segmentation performance in our experiments, and consider potential implications and future directions for this work.

II. METHOD

A. Tensor-based Pupil Image Representation

A tensor is defined as a multilinear mapping over a set of vector spaces [6]. Tensors represent a higher-order generalization of vectors and are commonly utilized for multidimensional data representation. Tensors are widely used to represent videos, color images, and hyperspectral images [7]–[9] through various data arrangements. In order to ensure the clarity of the notation in this study, we first present the notation and terminology related to tensors used in this paper [10], [11].

1. An N^th - order tensor denoted as $𝓟$ is formally defined as an element of the tensor product of $N$ vector spaces, where the order $\mathrm{N}$ of a tensor is the number of dimensions and $𝓟\in {\mathbb{R}}^{I_1\times I_2\times \cdots \times I_N}$.

2. Fibers are the higher-order analog of matrix rows and columns. A fiber of tensor $𝓟$ can be determined by keeping an index, while fixing others. A third-order tensor $𝓟$ involves column, row, and tube fibers denoted as $p_{:ml}$, $p_{n:l}$, and $p_{nm:}$, respectively.

3. Slices are two-dimensional portions of a tensor. A slice is defined by fixing all indices but two. A third-order tensor $𝓟$ contains slices denoted as $P_{n::}$, $P_{:m:}$, and $P_{::l}$, which are called horizontal, lateral, and frontal slices, respectively.

4. A tensor $𝓟\in {\mathbb{R}}^{I_1\times I_2\times \cdots \times I_N}$ has $\mathrm{N}$ ways to transform its data into a matrix called mode-$k$ unfolding of tensor $𝓟$ or tensor matricization. It is denoted by $P_{\left(k\right)}$, where $k\in \{\mathrm{1,2},\dots ,N\}$.

5. The mode-$k$ product is the result of the multiplication of a tensor by a matrix in mode-$k$. The mode-$k$ product $𝓟{\times }_{k}U$ of a tensor $𝓟\in {\mathbb{R}}^{I_1\times I_2\times \cdots \times I_N}$ with a matrix $U\in {\mathbb{R}}^{J\times I_k}$ is a tensor of size $\left(I_1\times \cdots \times I_{k-1}\times J\times I_{k+1}\times \cdots \times I_N\right)$. Therefore, we have

$${(𝓟{\times }_{k}U)}_{i_1\cdots i_{k-1}j{i}_{k+1}\cdots i_N}=\sum _{i_k}^{I_N}{x}_{i_1{i}_2\cdots i_N}{u}_{j{i}_k}.$$ (1)

In this study, to effectively exploit the relationship among color channels and spatial information, we construct a third-order tensor with three frontal slices denoting three color channels of a given color pupil image. In particular, given a color pupil image denoted $I\in {\mathbb{R}}^{h\times w}$ of size $h\times w$, it is split into $I^b$, $I^g$, and $I^r$, which correspond to the three-color channel images in RGB color space. A third-order tensor denoted as $𝓟\in {\mathbb{R}}^{h\times w\times 3}$ sized $h\times w\times 3$ is then constructed by adopting each color channel $I^c$, $c\in \{b,g,r\}$ to be a frontal slice $P_{::l}$, as depicted in Fig. 2.

Fig. 2.

Tensor-based pupil image construction. (a) A given color pupil image $I\in {\mathbb{R}}^{h\times w}$; (b) Blue, green, and red color channels $I^c$, $c\in \{b,g,r\}$ of (a); and (c) The third-order tensor $𝓟\in {\mathbb{R}}^{h\times w\times 3}$ of (a).

B. High-Order Singular Value Decomposition of a Color Pupil Image

A given $N$-order tensor $𝓟\in {\mathbb{R}}^{I_1\times I_2\times \cdots \times I_N}$ can be expressed as a linear combination of the outer products in different modes. Specifically, the high-order singular value decomposition (HOSVD) expresses tensor $𝓟\in {\mathbb{R}}^{I_1\times I_2\times \cdots \times I_N}$ as follows:

$$𝓟=\mathit{S}{\times }_1{U}^{\left(1\right)}{\times }_2{U}^{\left(2\right)}{\times }_3\cdots {\times }_N{U}^{\left(N\right)}.$$ (2)

It can also be expressed elementwise as:

$$p_{i_1{i}_2\cdots i_N}=\sum _{j_1=1}^{I_1}\sum _{j_2=1}^{I_2}\cdots \sum _{j_N=1}^{I_N}{s}_{j_1{j}_2\cdots j_N}{u}_{i_1{j}_1}^{\left(1\right)}{u}_{i_2{j}_2}^{\left(2\right)}\cdots u_{i_N{j}_N}^{\left(N\right)},$$ (3)

where $𝓢\in {\mathbb{R}}^{I_1\times I_2\times \cdots \times I_N}$ denotes the core tensor, which has the same size with the given tensor $𝓟$. $U^{\left(k\right)}\in {\mathbb{R}}^{I_k\times I_k}$, $k=\{\mathrm{1,2},\cdots ,N\}$ are matrices containing the left singular vectors of the mode-$k$ unfolding matrices $P_{\left(k\right)}$ of tensor $𝓟$. These matrices $U^{\left(k\right)}$ are also called inverse factors and determined through singular value decomposition of mode-$k$ unfolding matrices $P_{\left(k\right)}$ as follows:

$$P_{\left(k\right)}=U^{\left(k\right)}\sum ^{\left(k\right)}{V}^{\left(k\right)T},$$ (4)

where ${\mathrm{\Sigma }}^{\left(k\right)}$ denotes the singular value matrices and $V^{\left(k\right)}$ contains the right singular vectors of $P_{\left(k\right)}$. Furthermore, the core tensor denoted $𝓢$ is determined by:

$$𝓢=𝓧{\times }_1{U}^{\left(1\right)T}{\times }_2{U}^{\left(2\right)T}{\times }_3\cdots {\times }_N{U}^{\left(N\right)T}.$$ (5)

As with the given tensor, $𝓟$, mode-$k$ unfolding matrices $S_{\left(k\right)}$ of the core tensor $𝓢$ can be obtained by:

$$S_{\left(k\right)}={\mathrm{\Sigma }}^{\left(k\right)}{V}^{\left(k\right)}\left(U^{\left(N\right)}\otimes \cdots \otimes U^{(k+1)}\otimes U^{(k-1)}\otimes \cdots \otimes U^{\left(1\right)}\right),$$ (6)

where $\otimes$ denotes the Kronecker product of two matrices. At the end, a mode-$k$ unfolding matrix $P_{\left(k\right)}$ of the given tensor $𝓟$ can be reconstructed by:

$$P_{\left(k\right)}=U^{\left(k\right)}{S}_{\left(k\right)}{\left(U^{\left(N\right)}\otimes \cdots \otimes U^{(k+1)}{U}^{(k-1)}\otimes \cdots \otimes U^{\left(1\right)}\right)}^T.$$ (7)

In this study, we first construct a third-order tensor $𝓟\in {\mathbb{R}}^{h\times w\times 3}$ of a given pupil image $I\in {\mathbb{R}}^{h\times w}$ as discussed in subsection II-A. By using HOSVD, $𝓟$ is then decomposed into multiple components: three inverse factors denoted $U^{\left(1\right)}$, $U^{\left(2\right)}$, and $U^{\left(3\right)}$ and a core tensor $𝓢$, as depicted in Fig. 3. HOSVD decomposition is briefly presented in Algorithm 1.

Fig. 3.

The results of HOSVD decomposition of a third-order tensor $𝓟\in {\mathbb{R}}^{n\times w\times 3}$ in Fig. 2(c). (a) Inverse factor $U^{(1)}$; (b) Inverse factor $U^{(2)}$; (c) Inverse factor $U^{(3)}$; and (d) Core tensor $𝓢$.

Algorithm 1:

HOSVD

Input: A Color Pupil Image $I\in {\mathbb{R}}^{h\times w}$
Output: $U^{\left(1\right)}$, $U^{\left(2\right)}$, $U^{\left(3\right)}$, and $𝓢$
1.	$I^c,c\in \{b,g,r\}\leftarrow \mathrm{S}\mathrm{p}\mathrm{l}\mathrm{i}\mathrm{t}\left(I\right)$;
2.	$𝓟\in {\mathbb{R}}^{h\times w\times 3}\leftarrow \mathrm{R}\mathrm{e}\mathrm{s}\mathrm{h}\mathrm{a}\mathrm{p}\mathrm{e}\left(I^c\right)$;
3.	$get(U^{\left(k\right)},{\mathrm{\Sigma }}^{\left(k\right)},V^{\left(k\right)})$ via Eq. (4);
4.	$get\left(S_{\left(k\right)}\right)$ via Eq. (6);
5.	$𝓢\leftarrow \mathrm{R}\mathrm{e}\mathrm{s}\mathrm{h}\mathrm{a}\mathrm{p}\mathrm{e}\left(S_{\left(k\right)}\right)$
6.	Return $U^{\left(1\right)}$, $U^{\left(2\right)}$, $U^{\left(3\right)}$, $𝓢$.

C. Tensor-based Pupil Feature Extraction

In this study, we investigated the information carried by elements of inverse factor $U^{\left(3\right)}$ and sought to emphasize elements of $U^{\left(3\right)}$ seen to carry pupil-related information and de-emphasize elements seen to carry noise and background data. The feature-extracted image was then obtained by inverting HOSVD with the original $U^{\left(1\right)}$ and $U^{\left(2\right)}$ and a modified $U_m^{\left(3\right)}$.

The results of the mentioned analysis are depicted in Fig. 4. As seen, three elements in the first column of $U^{\left(3\right)}$ can reconstruct much of the three color channels, which are blue, green, and red, of the given input image. Accordingly, both pupil and interference information are mostly present and would be recovered from these elements. On the other hand, information specific to the pupil region appears to be present within the elements of the second column of the inverse factor $U^{\left(3\right)}$. As seen in Fig. 4, the third column of $U^{\left(3\right)}$ contains primarily interference information.

Fig. 4.

The information of Fig. 2(a) carried by nine elements of inverse factor $U^{(3)}\in {\mathbb{R}}^{3\times 3}$. The position of the image corresponds to the location of the investigated element in $U^{(3)}$.

In order to highlight the pupil region and eliminate the interference information in the given image, the elements of the first column were de-emphasized by subtracting off the mean of three elements. The proposed feature extraction algorithm is briefly presented in Algorithm 2 and the result of the proposed method is depicted in Fig. 5.

Fig. 5.

Result of the proposed pupil feature extraction based on tensor. (a) Original images from different cataract surgeries. (b) Feature images of (a) by the proposed method.

Algorithm 2:

TPFE

Input: Pupil Image $I\in {ℝ}^{h\times w}$
Output: Feature Image $F\in {ℝ}^{h\times w}$
1.	$\left\{U^{(1)},U^{(2)},U^{(3)},𝓢\right\}\leftarrow \mathrm{A}\mathrm{l}\mathrm{g}\mathrm{o}\mathrm{r}\mathrm{i}\mathrm{t}\mathrm{h}\mathrm{m}1(𝓟)$;
2.	$\mu \leftarrow \mathrm{m}\mathrm{e}\mathrm{a}\mathrm{n}\left\{U^{(3)}(\mathrm{1,1}),U^{(3)}(\mathrm{2,1}),U^{(3)}(\mathrm{3,1}\}\right\}$;
3.	$U^{(3)}(i,1)=(U^{(3)}(i,1)-\mu ),i\in \{\mathrm{1,2,3}\}$;
4.	$get\left(P_{(k)}^{(u)}\right),k\in \{\mathrm{1,2,3}\}$ via Eq. (7).
5.	${𝓟}^{(u)}=\left\{P_{::l,l=\mathrm{1,2},\cdots ,12}^{(u)}\right\}\leftarrow \mathrm{R}\mathrm{e}\mathrm{s}\mathrm{h}\mathrm{a}\mathrm{p}\mathrm{e}\left(P_{(k)}^{(u)}\right)$;
6.	$F^c,c\in \{b,g,r\}\leftarrow \mathrm{R}\mathrm{e}\mathrm{s}\mathrm{h}\mathrm{a}\mathrm{p}\mathrm{e}({𝓟}^{(u)})$;
7	$F\leftarrow \mathrm{M}\mathrm{e}\mathrm{r}\mathrm{g}\mathrm{e}\left(F^b,F^g,F^r\right)$;
8	Return $F$.

III. EXPERIMENTAL RESULTS AND DISCUSSIONS

A. Dataset

To evaluate the performance of the proposed method, we adopted a dataset consisting of 4,560 intraoperative images of size 480×270 pixels from 152 cataract surgeries, which were performed by experienced surgeons at Kellogg Eye Center, University of Michigan from 2020 to 2021. The pupil region on the images was manually annotated using MATLAB version R2022a with the Wacom One drawing tablet.

B. Pupil Recognition by Deep Learning Models

In order to assess performance of the proposed framework in improving the accuracy of pupil recognition, we adopted six state-of-the-art deep learning models: Xception [12], HR-Net [13], DeepLabV3+ [14], FPN [15], UNet [16], and LinkNet [17], each with VGG16 [18] as the backbone network for our experiments. The selected models were all pre-trained on ImageNet dataset [19] to reduce training time.

The dataset was randomly split into training and validation subsets consisting of 75% and 25% of the dataset, respectively. Thus, 3,420 images from 114 videos were included for training, 1,140 images from 38 videos were included for validation. Separate instances of each deep learning model were trained using either the original images or the feature images. A batch size of 8 and a learning rate of 10⁻⁴ were fixed for training. We trained each for a maximum of 100 epochs corresponding to 42,700 training iterations. To this end, precision, recall, Intersection over Union, and Dice coefficient of all six models on the validation set were compared with and without use of the feature-extracted images. The results of the experiments are shown in Table II. As seen, the accuracy of the deep learning models is considerably improved when combined with the proposed feature extraction TPFE method. Each of the 6 models tested demonstrated a statistically significant improvement in performance (p << 0.05 on Wilcoxon [20] signed rank testing) when utilizing the method.

TABLE II.

SEMANTIC SEGMENTATION RATE (%) OF DEEP LEARNING MODELS ON VALIDATION SET

Network Architecture	Precision (%)	Recall (%)	IoU (%)	Dice (%)
Xception	99.28	81.92	81.75	89.29
Xception + TPFE	98.97	85.71	85.37	91.50
HR-Net	99.24	86.88	86.72	92.36
HR-Net + TPFE	99.31	89.36	89.16	93.84
DeepLabV3+	99.37	86.71	86.58	92.39
DeepLabV3+ + TPFE	99.13	89.49	89.21	93.95
FPN	99.38	88.78	88.62	93.53
FPN + TPFE	99.45	88.97	89.20	93.88
UNet	99.33	87.79	87.62	92.96
UNet + TPFE	99.20	89.01	88.79	93.58
LinkNet	99.35	88.35	88.18	92.30
LinkNet + TPFE	99.10	89.50	89.21	93.83

IV. CONCLUSION

In this paper, we have proposed a novel feature extraction method named TPFE for pupil recognition systems in cataract surgery using deep learning models. The effectiveness of TPFE was comprehensively demonstrated through a set of experiments using state-of-the-art deep learning models. The results reveal that the accuracy of pupil recognition systems in cataract surgery can be significantly improved through utilization of the proposed TPFE method.