High-Tech Parameters for the Evaluation of Signs and Symptoms of Dry Eye Disease: Identification of Clinical Cut-Offs and Agreement with Low-Tech Tests

Abstract

Introduction

High-tech devices for the assessment of dry eye disease (DED) are increasingly available. However, the agreement between high- and low-tech parameters has been poorly explored to date. Trying to fill these gaps, we conducted a post hoc analysis on a recently published retrospective study on patients with DED receiving both low- and high-tech (Keratograph^®) assessments, and treatment with different lubricating eyedrops.

Methods

Six clinical questions were defined by the authors, considering literature gaps and their clinical experience, namely: (1) are NIKBUT-i and T-BUT interchangeable parameters? (2) What was the correlation between low- and high-tech parameters in untreated and treated patients with DED? (3) What was the correlation between signs and symptoms at baseline and during/after treatment? (4) Which parameters were better associated with symptoms? And with symptoms change over time? (5) What was the performance of NIKBUT-i and T-BUT in detecting clinically relevant changes? (6) What was the clinical advantage of adding other high- and low-tech parameters, respectively, to NIKBUT-i and T-BUT?

Results

Low-tech measures were the best descriptors of the Ocular Surface Disease Index (OSDI) at baseline. In contrast, high-tech assessments demonstrate better performance in detecting changes over time. The distribution of NIKBUT-i data was more dispersed than TBUT both at baseline and follow-up. At a fixed specificity of 80%, the sensitivity in detecting clinically relevant ameliorations of symptoms was 42% for NIKBUT-i and 25% for T-BUT. A battery of high-tech tests could detect 90% of clinical amelioration, compared with 45% with low-tech tests (p < 0.001). Correlation between low- and high-tech parameters in both treated and untreated patients is lacking.

Conclusions

Low-tech measures are adequate for diagnostic purposes in DED, whereas high-tech showed better performances at follow-up, particularly when different tests are combined. Overall, poor interchangeability among parameters and agreement with symptoms was reported both with high- and low-tech assessments.

Supplementary Information

The online version contains supplementary material available at 10.1007/s40123-024-01034-6.

Key Summary Points

Why carry out this study?

The agreement between high- and low-tech ophthalmological parameters has been poorly explored to date. Trying to fill the above-mentioned gaps, we conducted a post hoc analysis on the dataset of a retrospective study which evaluated the signs and symptoms of dry eye disease (DED) by means of low- and high-tech (Keratograph®) assessments in adult patients with DED owing to different causes, treated with different lubricating eyedrops.

What did the study ask?

This study aimed to investigate the agreement between different high- and low-tech parameters currently used to assess the signs and symptoms of DED and to define clinically relevant cut-offs to be considered in high-tech assessments.

Accordingly, some clinical questions were defined by the authors, considering literature gaps and their clinical experience.

What was learned from the study?

We provided one of the first comprehensive assessments of the agreement between high- and low-tech parameters in the evaluation of DED signs and symptoms.

The changes occurring during treatment are more easily detectable by means of high-tech devices, particularly when more tests are combined together.

Overall, poor interchangeability among parameters and agreement with symptoms was reported, suggesting the need to collect both signs and symptoms data during patient monitoring.

Introduction

Dry eye disease (DED) is a chronic, multifactorial disease of the ocular surface (OS) that poses a significant burden on global ocular health, and affects millions of individuals worldwide [1, 2]. Characterized by an array of signs and symptoms, including ocular discomfort, visual disturbances, and tear film instability, DED presents challenges in both diagnosis and management [3, 4].

Traditional diagnostic approaches rely heavily on subjective assessments and basic clinical tests, often overlooking nuances in disease presentation and progression. Among them, common clinical measures used for diagnosing DED are fluorescein tear breakup time (T-BUT), OS fluorescein staining, Schirmer test, and conjunctival hyperemia assessment [5, 6]. Overall, these low-tech assessments are limited by invasiveness (which affects the OS response and may lead to increased tearing reflex) and subjectivity. Moreover, different patterns of tear rupture can be seen in DED, making T-BUT difficult to interpret, with consequent low repeatability and reproducibility [7, 8].

In recent years, the introduction of different high-tech instruments has modified the clinical practice, providing objective and comprehensive evaluations of tear film quality, tear volume, and meibomian gland function [5, 9]. In particular, some of them (Keratograph 5M^®, IDRA^® Ocular Surface Analyzer, Tearcheck^®, LacryDiag Ocular Surface Analyzer) have the benefit of assessing multiple ocular surface parameters with a single device [De Luca 2023]. Among these options, Keratograph 5M^® is an all-in-one device used for the study of OS diseases, as it allows automated, non-eye contact evaluation of noninvasive Keratograph tear breakup time (NIKBUT), tear meniscus height (TMH), eyelid meibography, redness score, and conjunctival folds as a sign of conjunctivochalasis [8, 10].

Based on literature evidence, the use of these high-tech assessments is desirable for several reasons, such as the lack of contact with the patient and the independence from the operator, avoiding the use of dyes [5]. Moreover, it is important to remember that the current diagnosis of DED implies the assessment of tear film instability, and noninvasive devices should be preferred to fluorescein and T-BUT assessments [11]. At the same time, some limitations can be identified for high-tech assessments, such as the high variability, particularly in patients with photophobia, and the measurements limited to the central area of the cornea [8, 12–14].

The agreement between high-tech and low-tech parameters has been poorly explored to date, particularly in follow-up studies. However, when investigated, it is frequently low [13, 15, 16]. This is an expected finding, for example, for tear film instability, considering that high-tech instruments measure the central area of the tear film, whereas low-tech focuses on the whole cornea, and tear break-up is usually found in the inferior periphery of the cornea.

Recently, a retrospective study evaluated the signs and symptoms of DED by means of low- and high-tech assessments in a population of adult patients with DED owing to different causes, treated with different lubricating eyedrops [17]. Trying to fill the above-mentioned gaps, we used the dataset of that study to perform a post hoc analysis to investigate the agreement between different high- and low-tech parameters and to define clinically relevant cut-offs to be considered both in routine assessments of DED as well as in clinical studies.

Methods

Study Overview

This study aimed to investigate the agreement between different high- and low-tech parameters currently used to assess the signs and symptoms of DED and to define clinically relevant cut-offs to be considered in high-tech assessments. Accordingly, some clinical questions were defined by the authors, considering literature gaps and their clinical experience (see the following paragraph for details).

To respond to the clinical questions, we performed a post hoc analysis on the dataset of a previously published retrospective study that evaluated the signs and symptoms of DED by means of low- and high-tech assessments in a population of adult patients with DED owing to post-cataract surgery, meibomian gland dysfunction, allergy, or glaucoma medications [17]. Overall, the study included 155 patients who started treatment with different lubricating eyedrops (baseline, T0); follow-up assessments were carried out at 15 (T1) and 45 days (T2) [17]. High-tech assessment was performed by Keratograph 5M^® (K5M; Oculus Optikgerate GmbH, Wetzlar, Germany; distributed in Italy by Alfa Instruments SRL) for the measurement of NIKBUT (first or -i, average or -a, and class), tear meniscus height (TMH), eyelid meibography, redness score, and conjunctivochalasis; traditional low-tech measures were T-BUT, Schirmer test, conjunctival hyperemia (Efron scale), and corneal fluorescein staining. The Ocular Surface Disease Index (OSDI) score was also considered [17]. The study was conducted in accordance with the Declaration of Helsinki of 1964 and its later amendments. The inter-company Ethics Committee of Messina approved this study (Protocol number: 38–23). All the participants signed an informed consent form.

Clinical Questions

The following six clinical questions were defined:

1. Are NIKBUT-i and T-BUT interchangeable parameters?

2. What was the correlation between low- and high-tech parameters in untreated and treated patients with DED?

3. What was the correlation between signs and symptoms at baseline and during/after treatment?

4. Which parameters were better associated with symptoms? And with symptoms change over time?

5. What was the performance of NIKBUT-i and T-BUT in detecting clinically relevant changes?

6. What was the clinical advantage of adding other high- and low-tech parameters, respectively, to NIKBUT-i and T-BUT?

Statistical Analysis

The paired test analysis, chi-square test, Student’s t test, and analysis of variance were used to compare data, as appropriate. Histograms for data distribution were calculated. Relative standard deviation was used to compare the dispersion of the data as compared to their average. Pearson coefficients and Spearman rho were used to evaluate data correlations, as appropriate. To detect the possible association between symptoms and signs, a multiple regression analysis of raw data was performed at each visit, using the whole population and the subgroups of patients with different diagnoses. OSDI was used as the independent variable, and low- and high-tech as the dependent variables.

The correlation was calculated and classified according to the following scales: high, 0.81–1.00; substantial, 0.61–0.80; moderate; 0.41–0.60; fair, 0.21–0.40; negligible, 0.20–0.

The best descriptor of symptom change was intended as the parameter with the ability to detect a reduction of seven points or more in OSDI score, which was previously found to be associated with clinically relevant changes [18]. For NIKBUT and TBUT, ROC curves were used to select the best cut-off for discriminating clinical ameliorations.

Results

Are NIKBUT-i and T-BUT Interchangeable Parameters?

NIKBUT-i and T-BUT values distribution were different at T0, as shown in Fig. 1A. In particular, NIKBUT-i was significantly higher than T-BUT by a mean of 1.1 s [(range − 20 to 5); p = 0.004; Supplementary Table 1]. After starting treatment, NIKBUT-i was higher than T-BUT by 0.4 (− 20 to 13) at T1 and 0.5 (− 20 to 10) seconds at T2, but the difference was neither clinically nor statistically significant (Supplementary Table 1). The distribution of the differences between T-BUT and NIKBUT-i is reported in Fig. 1B.

Fig. 1.

NIKBUT-i and T-BUT distributions. A NIKBUT-i (red bars and curve) and T-BUT (blue bars and curve) values distribution in untreated patients with DED. B Distribution of the difference between T-BUT and NIKBUT-i (all patients with a difference lower than 10 s were grouped)

At T0, the percentage of patients showing a difference between T-BUT and NIKBUT-i of ± 0.5 s was 23%, and ± 1 s was 37%. During follow-up visits, the percentages were respectively 21% and 35% at T1 and 10% and 19% at T2. Only 33% of patients showing a difference between ± 0.5 s at baseline had the same difference at day 15, and 50% of the patients showing a difference between ± 1.0 s at baseline had the same difference at day 15.

At baseline, relative SD (RSD) was 76.1% for NIKBUT-i and 36.3% for T-BUT. Interestingly, the use of treatments was associated with a reduction of relative SD for NIKBUT-i (RSD of 63.9–53.7%) and an increase of the same parameter for T-BUT (RSD of 43.5–40.31%).

Overall, NIKBUT parameters (-I, -a, and class) were strongly associated, suggesting that these assessments do not provide an adjunctive advantage if studied separately, except in subgroup analysis.

What was the Correlation Between Low- and High-Tech Parameters in Untreated and Treated Patients with DED?

The correlation of NIKBUT-i and T-BUT was negligible both at T0 and at T1 and fair at T2 (p = 0.003, Table 1). The correlation between the Schirmer test and tear meniscus was negligible (Table 1), while the correlation between the redness score and Efron scale was substantial at baseline visit and after treatment (Table 1).

Table 1.

Correlation between low- and high-tech parameters during each visit

	T-BUT vs. NIKBUT-i	Schirmer vs. tear meniscus	Efron vs. redness score
T0	r = − 0.04 p = 0.616	r = 0.107 p = 0.198	r = 0.603 p < 0.001*
T1	r = 0.13 p = 0.110	r = 0.01 p = 0.856	r = 0.506 p < 0.001*
T2	r = 0.24 p = 0.003*	r = 0.01 p = 0.232	r = 0.476 p < 0.001*

*p < 0.05

What was the Correlation Between Signs and Symptoms at Baseline and During/After Treatment?

The correlation between signs and symptoms was negligible at all time points, except for a fair correlation between OSDI and T-BUT at T0 (r = − 0.28, p < 0.001, Table 2).

Table 2.

Correlation between T-BUT and OSDI, NIKBUT-i and OSDI for each visit

	T-BUT vs. OSDI	OSDI vs. NIKBUT-i
T0	r = − 0.28 p < 0.001*	r = − 0.01 p = 0.916
T1	r = − 0.18 p = 0.029*	r = − 0.03 p = 0.751
T2	r = − 0.07 p = 0.393	r = − 0.16 p = 0.047*

*p < 0.05

Which Parameters were Better Associated with Symptoms? And with Symptoms Change Over Time?

In the overall study population, a significant association was reported at T0 between OSDI score and both T-BUT (b = − 2.7, p < 0.001) and Schirmer test (b = 0.7, p < 0.001). At T1, significant results were found only for T-BUT (b = − 0.7, p = 0.033).

At T2, the best descriptors of the OSDI score were NIKBUT-i (b = − 0.5, p = 0.017) and the Schirmer test (b = − 0.3, p = 0.014). See Supplementary materials for details on multiple regression analysis (models 1, 2, 3).

In patients with allergies, the regression analysis was statistically significant for the Schirmer test (b = 0.7, p = 0.016). At T1 and T2, the regression analysis was not statistically significant (see Supplementary materials for details on multiple regression analysis, models 4, 5, 6).

In patients with blepharitis, the regression analysis was statistically significant for corneal staining (b = − 9.0, p = 0.027). At T1, the regression analysis was statistically significant only for the Efron scale (b = − 6.8, p = 0.038). On day 45, the best descriptor of the OSDI score was meibography (b = − 7.2, p = 0.035; see Supplementary materials for details on multiple regression analysis, models 7, 8, 9).

In patients with DED due to cataract surgery, the regression analysis was statistically significant for the Schirmer test (b = − 1.2, p = 0.005). At T1, the regression analysis was statistically significant for T-BUT (b = − 0.7, p = 0.030). At T2, the best descriptor of the OSDI score was NIKBUT-a (b = − 0.7, p = 0.003; see Supplementary materials for details on multiple regression analysis, models 10, 11, 12).

The correlation between changes in OSDI score and changes in any low- or high-tech parameter was negligible in all cases. Data for NIKBUT-i and TBUT are reported in Table 3.

Table 3.

Correlation between delta OSDI and delta NIKBUT-i and delta T-BUT

	Delta OSDI vs. delta NIKBUT-i	Delta OSDI vs. delta T-BUT
T1	r = 0.06 p = 0.493	r = 0.02 p = 0.828
T2	r = 0.07 p = 0.414	r = 0.03 p = 0.756

What was the Performance of NIKBUT-i and T-BUT in Detecting Clinically Relevant Changes?

The clinically significant improvement of OSDI (seven points or more) occurred in 71 patients at T2 (48.6%). This result was used to generate ROC curves for NIKBUT-i and T-BUT. The area under the curve (AROC) was 0.570 for NIKBUT-i and 0.530 for T-BUT (Fig. 2A, B). At a fixed specificity of 80%, the sensitivity was 42% for NIKBUT-i and 25% for T-BUT. The best cut-off was 2.0 s for NIKBUT-i and 4.0 s for T-BUT. If a cut-off of 2.0 s for T-BUT is chosen (as commonly done in clinical settings), this would correspond to a sensitivity of 53% and specificity of 47%. Assuming a specificity of 80%, a sensitivity of about 25% would be obtained.

Fig. 2.

ROC curves for NIKBUT-i (A) and T-BUT (B) cut-off determination

What was the Clinical Advantage of Adding Other High- and Low-Tech Parameters, Respectively, to NIKBUT-i and T-BUT?

Among the 71 patients with OSDI improvement of seven or more points at T2, 33 (47%) of them were correctly identified by NIKBUT-i > 2 s. When the assessment of the improvements of tear meniscus or redness score was added, 64 patients out of 71 (90%) were identified (Fig. 3). Regarding low-tech parameters, a T-BUT cut-off of > 4 s detected eight patients out of 71 with improved symptoms (11%), compared to 12 when corneal fluorescein staining reduction was added and 25 with the addition of conjunctival hyperemia reduction. Only 32 out of 71 patients (45%) were detected with any low-tech parameter (Fig. 4). The difference between high- and low-tech parameters in patient detection was significant (p < 0.001).

Fig. 3.

Venn diagram showing the detection of patients with clinically relevant amelioration using different high-tech parameters

Fig. 4.

Venn diagram showing the detection of patients with clinically relevant amelioration using different low-tech parameters

Discussion

The agreement between high- and low-tech parameters in the assessment of DED signs and symptoms has been poorly explored to date, particularly in follow-up studies [Cox 2015]. We conducted a post hoc analysis on a dataset of a retrospective study that evaluated the signs and symptoms of DED by means of low- and high-tech assessments in a population of adult patients with DED treated with different lubricating eyedrops [17]. Overall conclusive results of our study are summarized in Table 4.

Table 4.

Summary of overall results

Clinical questions	Answer after post hoc analysis
1. Are NIKBUT-i and T-BUT interchangeable parameters?	No, the two measures cannot be considered interchangeable both at baseline and under treatment In untreated patients, mean NIKBUT-i was higher than T-BUT by about 1 s, with a nearly twofold variability. In treated patients, the mean difference was 0.5 s, but only 20% of patients had a difference between NIKBUT-i and T-BUT lower than ± 1 s
2. What was the correlation between low- and high-tech parameters in untreated and treated patients with DED?	The correlation between low- and high-tech parameters was fair to negligible in both untreated and treated patients NIKBUT parameters (-I, -a and class) were strongly associated; thus, apparently do not provide adjunctive advantage if studied separately. We suggest to focus on the NIKBUT-i parameter
3. What was the correlation between signs and symptoms at baseline and during/after treatment?	At baseline, there was a fair inverse correlation between OSDI and T-BUT All the other correlations were negligible
4. Which parameters were better associated with symptoms? And with symptoms change over time?	The correlation between signs and symptoms was negligible both at baseline and under treatment, except for a fair correlation between OSDI and T-BUT at baseline At regression analysis, the parameters significantly associated with OSDI score were T-BUT and Schirmer test at baseline, T-BUT at 15 days and NIKBUT-i and Schirmer test at 45 days The correlation between changes in OSDI score and changes in any low- or high-tech parameter was negligible in all cases
5. What was the performance of NIKBUT-i and T-BUT in detecting clinically relevant changes?	The performance was overall superior with NIKBUT-i than with T-BUT. AROC was 0.570 for NIKBUT-i and 0.530 for T-BUT At a fixed specificity of 80%, the sensitivity was 42% for NIKBUT-i and 25% for T-BUT The best cut-off was 2.0 s for NIKBUT-i and 4.0 s for T-BUT
6. What was the clinical advantage of adding other high- and low-tech parameters, respectively, to NIKBUT-i and T-BUT?	High-tech parameters were superior to low-tech in detecting clinically relevant ameliorations. The detection rate was 47% for NIKBUT-i and 11% for T-BUT. Combining more tests, the undetection rate decreased to 10% for high-tech, and 55% for low-tech assessments. We recommend associating different high-tech parameters to improve patient management

According to the TFOS DEWS II, tear instability < 10 s is one of the diagnostic criteria of DED and should be preferably assessed noninvasively, i.e., through high-tech assessments [11]. Therefore, in our analysis, NIKBUT should be considered as the most relevant high-tech parameter.

In this study, we showed that NIKBUT parameters (-I, -a and class) were strongly associated among them, and they do not provide an adjunctive advantage if studied separately. We therefore focused on NIKBUT-i, which overall had values closer to TBUT. At baseline, NIKBUT-i and TBUT were significantly different, with NIKBUT-i obtaining a mean overestimation by 1.1 s. The difference is lower than reported by Wang et al. (2.0 s), and both studies confirm that the two assessments cannot be considered interchangeable in untreated patients with DED. NIKBUT-i also has a larger distribution than T-BUT (nearly twofold), and this is confirmative of previous findings [19].

During follow-up, under lubricating treatment, the difference between NIKBUT-i and T-BUT was reduced to a mean of 0.5 s, which was neither clinically nor statistically significant, indirectly suggesting that high-tech assessment could be used at follow-up without loss of consistency. Yet NIKBUT-i has a larger distribution than T-BUT also at follow-up, and the correlation between the two parameters remains low to moderate.

In line with previous experiences, our results showed the lack of correlation between low- and high-tech parameters in both treated and untreated patients, except for a fair correlation between NIKBUT-i and T-BUT at T2 (p = 0.003), probably due to the different approach at the basis of these measurements [15, 16]. However, it is interesting to note that low-tech measures (T-BUT and Schirmer) were the best descriptors of OSDI at baseline. In other words, our data support the use of low-tech assessments for diagnostic purposes, as done in most clinical settings. It has been claimed that high-tech instruments would have a superior discriminative ability in detecting DED [19]. At the same time, a recent literature review on DED and innovative diagnostic devices suggests that despite the consistent use of these tools in clinical settings could facilitate diagnosis, no diagnostic device can replace the TFOS algorithm up to date, further supporting our findings [De Luca 2023].

At follow-up, T-BUT loses its ability to describe OSDI. On the opposite, NIKBUT-i and the Schirmer test were the best descriptors at T2. Yet, the usability of Schirmer data is poor, as the mean change observed due to treatment was less than 2 mm/5 min.

NIKBUT-i was superior compared to T-BUT in correctly detecting patients showing a clinically relevant amelioration of symptoms, confirming the data from Wang et al. [19].

NIKBUT-i showed a twofold higher sensitivity than T-BUT at fixed 80%-specificity and a solid cut-off of 2.0 s to detect clinically meaningful OSDI change (compared to 4.0 s for T-BUT, very difficult to obtain in most patients).

Considering the absence of a correlation between OSDI change and change of signs, the diagnostic performance of NIKBUT in DED can be considered as lower than ideal. Our results suggest the possibility of further increasing its diagnostic power by adding other high-tech measures: only 7/71 ameliorated patients were not detected by a combination of high-tech measures, compared to 39/71 and 30/71 with low-tech assessments using the T-BUT cut-off of 4.0 and 2.0 s, respectively.

Our analysis presents some limitations, such as the retrospective nature of a single cohort of patients, which may limit the generalizability of the findings, and the short observation period, which may not fully capture long-term treatment effects. In addition, in this study, only one high-tech device was used, therefore the results obtained must be interpreted considering that the measurements could vary based on the use of other high-tech instruments, making it necessary in the future to carry out a comparison study between different high-tech technologies. At the same time, the value of our study relies on the first comprehensive assessments of the agreement between high- and low-tech parameters in the evaluation of DED signs and symptoms.

Conclusions

Given the results provided by this study, low-tech assessment resulted adequate for DED diagnosis. High-tech assessments showed better performance in detecting changes occurring over time due to treatments, and results are particularly superior to low-tech assessments when combining different tests. Overall, poor interchangeability among parameters and agreement with symptoms was reported, suggesting the need to collect both signs and symptoms data during patient monitoring. With regard to single patients' data, the correlation between low-, high-tech parameters and symptoms was overall negligible, reminding us of the complex, multifactorial and variable nature of DED.

Supplementary Information

Below is the link to the electronic supplementary material.

Author Contributions

Study design: Paolo Fogagnolo, Valentina Mirisola, Alfonso Strianese; data collection and interpretation: All; manuscript writing: Paolo Fogagnolo, Valentina Mirisola, Alfonso Strianese; manuscript editing: All; approval to submit: All.

Funding

There was no explicit funding for the development of this work. Editorial assistance and the journal’s Rapid Service Fee were supported by Alfa Intes.

Data Availability

The datasets generated during and/or analyzed during the current study are available from the corresponding author upon reasonable request.

Declarations

Conflict of Interest

All named authors confirm that they have no competing interests to declare.

Ethical Approval

The study was conducted in accordance with the Declaration of Helsinki of 1964 and its later amendments. The original study was conducted with the protocol approved by the inter-company Ethics Committee of Messina (protocol number 38-23). All the participants signed an informed consent form.