Meibomian glands segmentation in infrared images with limited annotation

Abstract

AIM

To investigate a pioneering framework for the segmentation of meibomian glands (MGs), using limited annotations to reduce the workload on ophthalmologists and enhance the efficiency of clinical diagnosis.

METHODS

Totally 203 infrared meibomian images from 138 patients with dry eye disease, accompanied by corresponding annotations, were gathered for the study. A rectified scribble-supervised gland segmentation (RSSGS) model, incorporating temporal ensemble prediction, uncertainty estimation, and a transformation equivariance constraint, was introduced to address constraints imposed by limited supervision information inherent in scribble annotations. The viability and efficacy of the proposed model were assessed based on accuracy, intersection over union (IoU), and dice coefficient.

RESULTS

Using manual labels as the gold standard, RSSGS demonstrated outcomes with an accuracy of 93.54%, a dice coefficient of 78.02%, and an IoU of 64.18%. Notably, these performance metrics exceed the current weakly supervised state-of-the-art methods by 0.76%, 2.06%, and 2.69%, respectively. Furthermore, despite achieving a substantial 80% reduction in annotation costs, it only lags behind fully annotated methods by 0.72%, 1.51%, and 2.04%.

CONCLUSION

An innovative automatic segmentation model is developed for MGs in infrared eyelid images, using scribble annotation for training. This model maintains an exceptionally high level of segmentation accuracy while substantially reducing training costs. It holds substantial utility for calculating clinical parameters, thereby greatly enhancing the diagnostic efficiency of ophthalmologists in evaluating meibomian gland dysfunction.

INTRODUCTION

The meibomian glands (MGs) are a series of large sebaceous glands that extend along the eyelid margin behind the eyelashes. They are responsible for producing and secreting lipids called blepharoplasts, which constitute the lipid layer of the tear film. This lipid layer is crucial for maintaining the health and integrity of the ocular surface and is an essential component of the tear film functional unit^[1]. Meibomian gland dysfunction (MGD) is a chronic and diffuse condition affecting MGs and is considered the primary cause of various ocular diseases, including dry eye and blepharitis^[2]. Epidemiological studies have demonstrated a global prevalence of MGD exceeding 35.9%, with variations among different racial groups ranging from 21.2% to 71.0%^[3]. Infrared imaging has emerged as an effective clinical technique for assessing the morphological characteristics of the MGs^[4]. Ophthalmologists employ these images to observe and analyze various MGs features, providing valuable insights for diagnosing MGD^[5]. However, relying solely on visual observation and clinical experience is subjective and yields relatively low reproducibility.

In response to the challenges mentioned above, Koh et al^[6] and Liang et al^[7] investigated methods for distinguishing healthy images from unhealthy ones, while Fu et al^[8] proposes a classification model based on color deconvolution and transformer structure. Furthermore, leveraging advancements in deep learning and medical image processing^[9] , Prabhu et al^[10] used the U-Net architecture for MG segmentation, while Khan et al^[11] introduced an adversarial learning-based approach to enhance segmentation accuracy. Lin et al^[12] introduced an improved U-Net++ model for MGs segmentation, yielding remarkable results. However, despite these advancements, many existing MGs segmentation approaches face limitations in clinical application due to their heavy reliance on a substantial amount of fully annotated label data. Manual annotation can be exceedingly time-consuming and labor-intensive, considering the considerable number and close arrangement of glands. Even for experienced ophthalmologists, the process can take an average of 5 to 8min per image^[13].

To overcome the constraints associated with fully annotated training datasets, researchers have embraced the concept of weakly supervised learning, aiming to alleviate challenges by leveraging sparsely annotated forms such as scribble annotation, box-level annotation, or image-level annotation. Notably, among these, scribble annotation has gained increasing popularity. An annotated scribble comprises a set of pixels with corresponding category labels, with unannotated pixels treated as unknown. Lin et al^[14] achieved the first successful scribble application in image segmentation. Lee et al^[15] proposed a method for generating pseudo-labels for cell segmentation with scribble annotation, demonstrating the significant supportive role of pseudo-labels in segmentation. Cao et al^[16] identified that unreliable pixels can substantially perturb the scribble network, emphasizing the necessity of distinguishing between reliable and unreliable pixels.

Building upon these approaches, this study investigates the use of scribble annotation to achieve satisfactory automatic segmentation of MGs in infrared images. The proposed model is anticipated to substantially alleviate the manual labeling burden while simultaneously maintaining segmentation performance. This advancement is expected to enhance the diagnostic efficiency of MGD and facilitate the broader application of medical image processing.

MATERIALS AND METHODS

Ethical Approval

This study received approval from the Ethics Committee of Fujian Provincial Hospital (K2020-03-124) and adhered to the principles outlined in the Declaration of Helsinki. All subjects were duly informed and consented to participate in this study.

Acquisition and Pretreatment of Infrared MG Images

The infrared MG images used in this study were graciously provided by the Department of Ophthalmology, Fujian Provincial Hospital. These images were captured using an ocular surface comprehensive analyzer (Keratograph 5 M). The dataset comprised 138 patients with dry eyes (276 eyes) from January 2020 to June 2021. Given the technical challenges associated with everting the lower eyelid, particularly because of the absence of a substantial tarsal plate compared with the upper eyelid, MGs images of the lower lid often exhibit uneven focus and partial exposure. Consequently, in alignment with established research practices^[17], our study concentrates exclusively on upper eyelid images. A batch cropping process was executed to eliminate potential confounding information, resulting in images with dimensions of 740×350 pixels (Figure 1A). The contrast enhancement mode was employed to accentuate the visualization of the MGs. After meticulous selection, images presenting issues such as eyelash occlusion, incomplete coverage, excessive blurriness, and substantial glare were excluded. Consequently, 203 images were retained.

Figure 1. Dataset of infrared MGs images.

A: Cropped image; B: Full annotation, meibomian (red) and the MGs (green); C: Input image; D: Scribble annotation, meibomian (red) and the MGs (green). MGs: Meibomian glands.

Construction of Dataset

Two datasets were meticulously developed for model training and validation: an MGs scribbled dataset and a fully annotated dataset. Preprocessed images underwent annotation by three experienced ophthalmologists, each possessing over one year of clinical experience. Annotation procedures were conducted using the polygon tool in the Labelme software, facilitating comprehensive delineation of the meibomian region and individual structures of the eyelid glands. In instances of annotation discrepancies, a fourth senior ophthalmologist was consulted to make the final adjudication. The fully annotated results are illustrated in Figure 1B. The meibomian region mask was multiplied with the cropped original image to eliminate interference, such as eyelashes, resulting in the final input image (Figure 1C). We processed the fully annotated dataset to construct the scribbled dataset to alleviate annotation pressure and draw inspiration from the skeletonization algorithm. The morphological skeletonization algorithm was employed to extract intermediate lines representing the foreground and background regions, simulating manually generated scribble annotations (Figure 1D). After thorough expert verification, the generated scribble annotations from our method exhibited no discernible difference from the manually annotated scribbles by experts. The dataset used in this study comprises 203 sets of images, partitioned into training, validation, and test sets with a ratio of 7:1:2. The training and validation sets were employed for model training and parameter tuning. In contrast, the test set was used to validate the proposed method.

Model for MGs Segmentation in Eyelids

Within the framework of weakly supervised learning, we introduce an rectified scribble-supervised gland segmentation (RSSGS) model (Figure 2). This model uses the U-Net architecture as the segmentation network and integrates three key strategies: temporal ensemble prediction (TEP), uncertainty estimation, and transformation consistency constraint. Each image x, paired with its corresponding scribble label, serves as input to the model. The proposed network is trained with cross-entropy loss for scribble pixels as follows:

Figure 2. Framework of the proposed method.

$$L_{sp}={\displaystyle \sum _{c=1}^M{\displaystyle \sum _{i\in \Omega s}{S}_{c,i}\log P_{c,i}}}$$ (1)

Where S_c,i and P_c,i represent the scribbled pixel and the model's predicted probability for the i pixel in the c class, respectively. Ωs denotes the set of scribble pixels. |Ωs| and M represent the total number of scribble pixels and classes, respectively.

For unlabeled pixels, we utilized the predicted exponential moving average (EMA) to generate pseudo-labels, offering initial segmentations for the unlabeled regions based on the model's predictions.

Recognizing the potential disruption caused by unreliable labels, we integrate uncertainty estimation, quantifying the confidence level associated with each pseudo-label. Furthermore, we introduced a transformation consistency constraint strategy to enforce uniformity in the model's predictions, resulting in more reliable and accurate segmentation.

For unlabeled pixels, we utilized the predicted EMA to generate pseudo-labels, offering initial segmentations for the unlabeled regions based on the model's predictions. Recognizing the potential disruption caused by unreliable labels, we integrate uncertainty estimation, quantifying the confidence level associated with each pseudo-label. Furthermore, we introduced a transformation consistency constraint strategy to enforce uniformity in the model's predictions, resulting in more reliable and accurate segmentation.

Temporal Ensemble Prediction

To address the lack of supervision for unlabeled pixels, we drew inspiration from Scribble2Label^[15] and introduced the TEP strategy. We used EMA^[18] technique to amalgamate historical and current predictions throughout our network training process. This process results in an ensemble prediction encompassing knowledge learned from previous iterations, as the formula equation (2) indicates.

$$y_n=\alpha p_i+(1-\alpha )y_{n-1}$$ (2)

where y represents the average prediction value, y_o=p₁. α is the adaptive ensemble momentum (refer to equation 4), n denotes the number of average predictions. To ensure efficient integration and reduce computational expenses, we perform pseudo-label updates only every γ cycle and set the value of γ to 5.

Uncertainty Estimation

One limitation of the TEP strategy is that it treats reliable and unreliable predictions equally when generating pseudo-labels. However, the presence of unreliable predictions can have a negative impact on network optimization. Therefore, it is crucial to mitigate the effects of these unreliable predictions.

Inspired by uncertainty theory^[19], we introduced an uncertainty graph that evaluates the uncertainty in the model predictions. The uncertainty graph can be represented as follows:

$$U=\frac{1}{K}{\displaystyle \sum _{k=1}^K{\left(p^k\right)}^2}-{\left(\frac{1}{K}{\displaystyle \sum _{k=1}^K{p}^k}\right)}^2$$ (3)

where K controls the number of times, the model processes each image during training and is set to two in our approach. P^k represents the model output for the k-th pass. Introducing random Gaussian noise to the input images induces variability in the model's output for each pass, facilitating uncertainty graph estimation.

Subsequently, we introduce an adaptive ensemble momentum mapping that varies for each pixel on the basis of the uncertainty graph U to guide the generation of pseudo-labels. This momentum mapping α is defined as follows:

$$\alpha =\lambda (1-U)+U$$ (4)

where λ=0.6. By incorporating the adaptive weight α, we dynamically adjust the influence of each pixel's prediction on the basis of its associated uncertainty. This enables us to prioritize certain predictions while down-weighting the contributions of uncertain predictions during pseudo-label update.

Additionally, we define a confidence threshold for label filtering. This threshold enables us to remove unreliable predictions and retain those deemed more reliable for network optimization. The label filtering function is represented by equation (5):

$$f(x)=\{\begin{array}{l}z,max(z,1-z)>\tau \\ \delta ,other\end{array}$$ (5)

where τ represents the confidence threshold set at 0.8, δ denotes the grayscale value of unlabeled pixels, and z represents the grayscale value of scribbled pixels.

Having obtained with the reliable pseudo-labels in hand, we optimize the unlabeled pixels. The loss function for unlabeled pixels is defined in equation (6).

$$L_{up}=-\frac{1}{\left|{\Omega }_u\right|}{\displaystyle \sum _{c=1}^M{\displaystyle \sum _{i\in {\Omega }_u}f\left(z_{c,i}^n,\tau \right)}}log{P}_{c,i}$$ (6)

where Z_c,i and P_c,i represent the pseudo-label and model prediction probability for the i-th pixel in the c-th class. Ω_u denotes the set of unlabeled pixels, and |Ω_u| represents the total number of unlabeled pixels. The L_up also excludes unlabeled pixels in the pseudo-labels. Due to the initially generated unreliable pseudo-labels, L_up is applied for model optimization after the E-th epoch, which is set to 100 in our approach.

Transformation Equivariance Constraint

In weakly supervised learning, it is expected that applying specific transformations such as flip, rotate, and scale to an input image will yield an equivariant segmentation outcome. This property, known as transformation consistency, is shown in equation (7). This consistency is crucial for maintaining spatial coherence and stability in the model's predictions, even with limited annotated data.

$$F(T(I))=T(F(I))$$ (7)

where I is the input image, T is the transform function, and F is the segmentation network.

We impose constraints on these transformations through cosine similarity loss L_tc as follows:

$$L_{tc}=\frac{F(T(I))*T(F(I))}{{\left|\left|F(T(I))\right|\right|}_2*{\left|\left|T(F(I))\right|\right|}_2}$$ (8)

where ||·|| denotes the L2 paradigm.

The RSSGS framework optimizes the model using a weighted sum of three losses: the scribble pixel loss (L_sp), the unlabeled pixel loss (L_up), and the transformation consistency constraint (L_tc), as shown in equation (9):

$$L_{total}=L_{sp}+{\lambda }_u{L}_{up}+{\lambda }_t{L}_{tc}$$ (9)

Here, λ_u=0.5, and the parameter λ_t represents a weight function, as illustrated in equation (10). This weight function dynamically adjusts the magnitude of the transform consistency loss based on the number of training cycles. This adaptive approach serves the purpose of alleviating the premature impact of transform consistency on the network.

$${\lambda }_t=1.0*e^{-5{\left(1-\frac{t}{t_{\max }}\right)}^2}$$ (10)

where t denotes the current number of training cycles, while t_max represents the maximum number of training cycles.

RESULTS

Adhering to the criteria outlined in pertinent literature^[20], we employ accuracy (Acc), dice coefficient (Dice), and intersection over union (IoU) as evaluation metrics for our model. The training regimen spans 300 epochs across all datasets, using a batch size of eight. The RAdam optimizer is employed with an initial learning rate of 0.0003. To bolster the credibility of our experimental outcomes, a 5-fold cross-validation approach was implemented. Our comparative analysis assesses our method against state-of-the-art techniques for medical image segmentation. This evaluation compares models trained with either full annotation or scribble annotation.

Segmentation Results

Figure 3 displays the segmentation outcomes of our method (Figure 3D) and the U-Net (full annotation) approach (Figure 3C) on MGs in infrared images. Both methods accurately segment MGs within non-edge regions, effectively capturing their irregular shapes. However, in edge regions, particularly in ambiguous areas, the U-Net^[21] exhibits some segmentation errors, whereas our approach maintains a commendable performance, possibly attributed to the reliable pseudo-label.

Figure 3. MGs segmentation results.

A: Input images; B: Ground-truth; C: U-Net (full supervision); D: Our method. MGs: Meibomian glands.

Outperformance Compared with Other Methods

We conducted a comparative analysis of our method against mainstream medical image segmentation approaches employing scribble labels. The methods considered include the minimization of the regularized loss method (PCE)^[22], the Gated CRF loss-based method (Gated CRF)^[23], the Mumford-Shah loss-based method (M-S)^[24], an efficient method for deep neural network (Pseudo-Label)^[25] dual-branch network method (Dual-branch Net)^[26], and Scribble2Label^[15].

In Table 1^[21]–^[28], our method excels with Acc, Dice, and IoU at 93.54%, 78.02%, and 64.18%, respectively. Outperforming other weak-supervised methods across all metrics, our method demonstrates improvements of 0.76% in Acc, 2.06% in Dice, and 2.69% in IoU compared with the leading Scribble2Label.

Table 1. Performance comparisons with other state-of-the-art methods.

Label	Method	Acc (%)	Dice (%)	IoU (%)
Full	U-Net[21]	94.26	79.53	66.22
	U-Net++[27]	94.41	79.76	66.47
	U-Net3+[28]	93.85	78.67	65.89
Scribble (100%)	PCE[22]	91.96	74.48	59.64
	Gated CRF[23]	92.32	75.64	61.12
	M-S[24]	84.34	60.27	43.94
	Pseudo-Label[25]	92.34	75.35	60.70
	Dual-branch Net[26]	92.14	75.30	60.68
	Scribble2Label[15]	92.78	75.96	61.49
	RSSGS	93.54	78.02	64.18

RSSGS: Rectified scribble-supervised gland segmentation; M-S: Mumford-Shah loss-based; PCE: Minimization of the regularized loss; Acc: Accuracy; Dice: Dice coefficient; IoU: Intersection of union.

In a broader comparison with models trained using full labels, including U-Net, skip-connections-based U-Net++^[27], and full-scale connected U-Net3+^[28], our approach, while slightly less effective than U-Net++, demonstrates reductions of 0.8% in Acc, 1.9% in Dice, and 3.1% in IoU. Importantly, our method achieves a significant 80% reduction in annotation costs, highlighting its capacity to minimize performance loss even with substantial cost cutting. In summary, our method consistently exhibits exceptional MG segmentation performance.

Ablation Study

In this section, we conduct an ablation study to analyze the roles of uncertainty estimation (L_up) and transformation equivariance constraint (L_tc). All experiments were conducted on scribble datasets, with U-Net used for evaluating the results. Table 2 presents the improvements associated with each component. The results indicate that the proposed method effectively enhance model performance when applied individually. Moreover, there is a synergistic effect when these methods are employed together. Specifically, the combined use resulted in an improvement of 1.58% in Acc, 3.54% in Dice, and 4.54% in IoU.

Table 2. Ablation study on the effectiveness of different components.

Lsp	Lup	Ltc	Acc (%)	Dice (%)	IoU (%)
√			91.96	74.48	59.64
√	√		93.27	77.16	63.08
√		√	92.41	75.70	61.17
√	√	√	93.54	78.02	64.18

Acc: Accuracy; Dice: Dice coefficient; IoU: Intersection of union.

DISCUSSION

The quality of the scribble annotations substantially influences the model's performance. To assess the robustness of our method, we conducted experiments using scribble annotations with varying quality levels. Specifically, we randomly retained 10%, 30%, 50%, 70%, and 90% of the scribble pixels and used them as input for the experiment. As depicted in Table 3, a diminishing trend in the model's performance was observed as the proportion of scribble annotations decreased. For instance, at a 10% proportion, the model achieved Acc, Dice, and IoU of 92.61%, 75.76%, and 61.23%, respectively. These values surpassed those of many methods employing 100% proportion, with only marginal decreases of 0.93%, 2.26%, and 2.88% compared with the results obtained with 100% proportion. This observation underscores the effectiveness of the proposed method in maintaining high performance even with reduced scribble annotation proportions.

Table 3. Quantitative results with various amounts of scribbles.

Method	Scribble amounts	Acc (%)	Dice (%)	IoU (%)
RSSGS	10%	92.61	75.76	61.23
RSSGS	30%	93.06	76.70	62.41
RSSGS	50%	93.25	77.02	62.84
RSSGS	70%	93.23	77.08	62.92
RSSGS	90%	93.53	77.69	63.71
RSSGS	100%	93.54	78.02	64.18

RSSGS: Rectified scribble-supervised gland segmentation; Acc: Accuracy; Dice: Dice coefficient; IoU: Intersection of union.

Our method demonstrates exceptional performance in addressing the challenging task of MGs segmentation with sparse annotation. Outperforming the latest weakly supervised segmentation method, it achieves segmentation results comparable with those of fully supervised methods while substantially reducing annotation costs. Notably, our method even surpasses in specific details. Its robustness is evident when faced with varying levels of scribble annotation; even under a 10% setting, the achieved Acc, Dice, and IoU metrics only marginally decreased by 0.93%, 2.26%, and 2.88%, respectively, compared to the 100% setting. Confronted with the limited supervision challenge arising from a scarcity of labeled data, we meticulously address the unique aspects of infrared MG images, such as the glands' elongated and densely distributed characteristics. We introduce a temporal ensemble-based approach for generating pseudo-labels to augment label information, incorporating uncertainty estimation and consistency constraints to enhance label reliability. Our method effectively compensates for the limited information provided by scribble annotation, facilitating the automation of infrared MGs image analysis. This, in turn, provides valuable assistance to ophthalmologists in making accurate diagnoses.

Furthermore, a time efficiency test was conducted, engaging three experts to annotate 100 MG images using the scribble annotation method. The results revealed that the average time spent on scribble annotation (100%) for one image was approximately 1.2min. This starkly contrasts with the average 5–8min required to thoroughly label each image, indicating a substantial reduction in annotation burden.

Despite the promising outcomes, our study does have some limitations. First, our investigation was exclusive to the infrared MGs dataset. It is essential to recognize that the applicability of our approach extends beyond this dataset alone. In future research, we aim to explore the performance of our approach on a more diverse range of datasets, evaluating its effectiveness in clinical practice and fostering a deeper integration of deep learning and medical imaging, ultimately enhancing artificial intelligence's capacity to serve humanity more effectively. Second, concerning segmentation results, morphological parameters can be computed to derive various characteristics of individual glands, such as gland drop, tortuosity, and total gland count. Using these morphological parameters can offer valuable support to ophthalmologists in making accurate clinical diagnoses and designing appropriate treatment plans.

In conclusion, this study successfully addressed the segmentation task with limited labeled data, demonstrating remarkable performance. It promises substantial benefits for both physicians and patients.