Repeatability, reproducibility and agreement between three different diagnostic imaging platforms for tear film evaluation of normal and dry eye disease

Abstract

Purpose

To assess the repeatability and reliability of different commercially available diagnostic platforms in the objective assessment of tear film parameters and if there exists any agreement between them.

Methods

Thirty healthy individuals (N = 60 eyes) and fifteen DED patients (N = 30 eyes) had their tear film parameters (Lipid layer thickness (LLT), Tear meniscus height (TMH), Non-invasive tear break up time (NIBUT)) assessed using three instruments - LipiView® II, IDRA ocular surface analyser (IDRA-OSA) and Oculus keratograph 5 M (K5M). Bland-Altman analysis and linear mixed effects modelling & Generalized Linear Hypothesis Test were used for analysis and coefficient of variation (CoV).

Results

There is poor repeatability but good reproducibility of LLT values measured with Lipiview, or IDRA. NIBUT using K5M & IDRA-OSA shows good repeatability and reproducibility in control group but poor repeatability in DED patients. TMH values obtained with K5M or IDRA-OSA had poor repeatability with high CoV. Between two observers, good reproducibility is observed for TMH and NIBUT values using both K5M & IDRA-OSA but not for LLT values. Between instruments, all the measurements (LLT, NIBUT and TMH) were significantly different on Bland Altman analysis.

Conclusion

No two dry eye diagnostic platforms can be used interchangeably and non-invasive tear film values should be interpreted keeping in mind the individual machine’s variability. The high coefficient of variation in DED patients compared to normal reflects inherent variability in tear film irrespective of the device used.

Introduction

Diagnosis of dry eye disease (DED) relies on a battery of tests assessing tear film, corneal staining and tear volume [1]. As the symptomatology is similar across various aetiologies of DED, the accuracy of objective testing is of paramount importance in treating patients appropriately. Existing clinical tests such as tear break up time (TBUT) evaluation and Schirmer test have large examiner-based differences with inaccuracies in tear film height and inability to assess lipid layer. Lipid layer produced by meibomian glands maintains tear stability and its thickness correlates with gland loss [2].

Technological developments have considerably advanced for measuring tear film thickness, its constituents and stability in a non-invasive and reliable manner. The available platforms use either interferometry or videokeratographic principle. It is important to know the accuracy levels of these instruments and inter-instrumental variability. The Diagnostic Methodology Subcommittee of the International Dry Eye Workshop has stressed the importance of developing objective methods of tear film examination which will improve comparability of measurements [1]. Currently available techniques are automated and examiner independent for measuring non-invasive tear break up time (NIBUT), lipid layer thickness (LLT) and tear meniscus height (TMH). There are various instruments available in the market and these have not been compared yet, though few individual machine-based data for specific parameters are available [3–8]. Individually, the Oculus keratograph 5 M (K5M) has shown variable (low or good) repeatability for NIBUT values [4, 6] and moderate reproducibility for TMH values [9]. The reproducibility values are not available for IDRA ocular surface analyser. The current study aims to evaluate intraobserver repeatability, interobserver reproducibility and agreement in tear film parameters obtained using commercially available LipiView® interferometer, IDRA ocular surface analyser and K5M.

Methods

Subjects

This study followed the tenets of the Declarations of Helsinki and was approved by Institutional Ethics Committee, L V Prasad Eye Institute, Hyderabad, India. Consecutive patients, diagnosed with DED visiting the ocular surface clinic at a tertiary referral eye institute formed study group. Healthy volunteers without DED were taken as control group. All subjects, including healthy volunteers, had a comprehensive ophthalmic examination. All healthy controls had no dry eye symptoms and had normal Schirmer values (>10 mm) with no ocular surface staining [10]. Patients were diagnosed as having DED when a combination of dry eye symptoms along with ocular surface disease index (OSDI) score >13, Schirmer score <10 mm or NIBUT < 7 s, and positive corneal staining (>1 on oxford staining score) were present. The excluded individuals were those aged <18 years, contact lens wearers, those who underwent refractive surgery, or those with prior ocular surgery within past 3 months, ocular injury, or other ocular diseases such as ocular infection or glaucoma or retinal problems.

The OSDI questionnaire was completed for every participant. Two investigators (SSA and ZM) operated the instruments and a single measurement when labelled suitable by the instrument was performed. The measurements were obtained on different days between 10 am and 4 pm in a dimly lit room with controlled temperature (23–25 °C) and humidity (30–40%). One examiner (ZM) repeated the measurements in all subjects for assessing intraobserver repeatability. No two readings on the same subject were separate by more than one week. The second observer performed measurements in the same room with relatively similar environmental conditions on the same day. There was randomisation of the eye to be tested first as well as the sequence of the device use. There was an interval of 5 min between measurements. The other eye was tested 20 s after the first while evaluating NIBUT.

Devices used in the study

Table 1 highlights the differences between these devices. The LipiView® II Ocular Surface Interferometer (TearScience® Inc, Morrisville, NC, USA) measures the absolute value of LLT by studying a billion data points of the interferometric image of the tear film. The technique of Lipiview interferometer was similar to that used in the study of Eom et al. [2]. The participants maintained fixation of the internal target while being allowed to blink normally, and a video was captured for 20 s. The camera focus was adjusted for a clear interferometric image focused on tear film plane and the pupil lied in the centre of the live video screen. LLT is processed in interferometric colour units (ICU) where 1 ICU corresponds to ~1 nm.

Table 1.

Product details of three dry eye diagnostic devices used in the current paper.

	LipiView II Interferometer	Keratograph 5 M	IDRA Ocular surface analyser
Year of introduction	2011	2012	2018
Size of machine
Weight	52 lbs	7.1 lbs (head), 13.5 lbs (base)	4.6 lbs
Dimensions	17 in. W x 31 in. D x 18 in. H	10.2 in. × 12.5 in. W – 15.7 in. × 19 in. D – 20.1 in. H	8 in. W* 10 D* 8 in. H
Principle of acquisition
Light source	Near infra-red, white light	Placido illumination- infra-red and white, LED- blue, white & infra-red	Infra-red LED- Blue, red and white LED
Lipid layer	Interference Colour Unit score calculated from more than 1billion data points of the interferometric image captured from lower half of the cornea	Colour interferometry of tear film- subjective assessment of colour intensity	White light reflected from the tear film in a white fan-shaped area covering the lower third of cornea. Automatic interference of colours is converted into thickness of lipid layer in nm.
Tear meniscus height	-	High resolution white/ infra-red photo of the ocular surface- manually select TMH	White light flash photo of the ocular surface- manually select TMH
Non-invasive tear break up time	-	Infra-red placido ring illumination Time taken for disruption of the reflected placido rings onto corneal surface- first and average values (automatic)	White light placido ring illumination Time taken for disruption of the reflected placido rings onto corneal surface- first and average values (automatic/manual mode)
Meibography	Near Infra-red illumination of everted eyelids	Infra-red LED spots for the meibomian gland scanning	Infra-red illumination of everted eyelids, Also, gives degree of drop out
Blink rate	Automatically detect and analyse blink rate	Can capture video and then calculate manually	Automatically detect and analyse blink rate
Other ocular surface features	-	Tear film dynamic video, Ocular redness, video imaging of ocular surface	Ocular redness, blepharitis assessment
Slit lamp attachment	No	No	No

The IDRA Ocular Surface Analyzer (SBM Sistemi, Coburn Technologies, Inc, Italy) is a fully automated, comprehensive dry eye diagnostic system that performs non-invasive interferometry, meibography, TMH measurement and auto-NIBUT testing. The average LLT, auto-NIBUT and TMH were measured using IDRA. For LLT, interferometric principle is used and the surface reflection pattern of coloured lipids is compared with an inbuilt grading scale. NIBUT is measured using grids that are projected onto cornea and time from the last complete blink to the first discontinuity in grid is measured as first NIBUT. It gives average NIBUT over the tested duration which was 14 s in our study. No manual NIBUT was calculated to avoid subjective error. TMH was measured at three points: central, and medial and lateral paracentral locations. The average of three readings was taken as the final TMH.

The Oculus® keratograph 5 M (K5M; Oculus®, Wetzlar, Germany) is a corneal topographer that uses infra-red light source to capture placido disc image and calculates the image irregularity as NIBUT. Firstly, the image centration was performed and then the participants were asked to blink twice. After last blink, subjects were instructed to look at a fixation light and keep eyes open as long as they can. If an individual blinked before 12 s, the test was repeated after a relaxation time of 10 min.

Statistical analysis

The statistical analysis was performed using the software R Core Team (R Foundation for Statistical Computing, Vienna, Austria). A mixed effects model applying maximum likelihood estimation was used to estimate summary measures and make comparisons between measurements. Coefficient of variation (CoV) was calculated for precision and repeatability data. Bland-Altman analysis was performed to estimate the mean agreement and the 95% limits of agreement. A p value of <0.05 is normally considered statistically significant. The p value is reported in Table 2 for Linear Mixed Effect modelling & Generalized Linear Hypothesis Test for repeatability. The high p value indicates that the mean difference within the observer is zero. A greater CoV value indicates that the level of dispersion around the mean is greater, hence less precise.

Table 2.

Summary of the tear film measurements of normal and DED patients.

Device	Observer	Measurement	Variable	Normal, Mean ± standard deviation	DED, Mean ± standard deviation
IDRA	1	I	LLT	47.35 ± 6.54	58.76 ± 13.99
		II		49.07 ± 7.00	63.06 ± 18.5
	2	I		53.92 ± 14.57	65.53 ± 11.36
LipiView	1	I	LLT	59.70 ± 17.21	62.1 ± 22.08
		II		58.03 ± 17.85	58.06 ± 23.25
	2	I		56.70 ± 16.47	60.3 ± 22.08
IDRA	1	I	TMH	0.23 ± 0.05	0.28 ± 0.08
		II		0.21 ± 0.06	0.30 ± 0.08
	2	I		0.20 ± 0.03	0.28 ± 0.06
Oculus	1	I	TMH	0.27 ± 0.07	0.23 ± 0.08
		II		0.27 ± 0.10	0.26 ± 0.10
	2	I		0.25 ± 0.03	0.24 ± 0.07
IDRA	1	I	NIBUT	9.00 ± 1.29	8.00 ± 3.8
		II		8.81 ± 1.64	7.49 ± 3.68
	2	I		8.43 ± 1.35	6.97 ± 1.84
Oculus	1	I	NIBUT	11.80 ± 2.19	9.20 ± 4.29
		II		12.05 ± 1.52	9.8 ± 4.83
	2	I		11.06 ± 2.36	8.74 ± 3.5

LLT lipid layer thickness, NIBUT non-invasive tear break-up time, TMH tear meniscus height.

Results

Demographics and clinical data

A total of 30 healthy volunteers (N = 60 eyes) with a mean age of 31 years (SD, 7.9) and 15 patients with DED (N = 30 eyes; mean age 43; SD,15) were included in the study. In the control group, there were 12 males and mean OSDI value was 4.2 (range, 0 to 6). None of them were contact lens users or were using any ocular medications. Slit-lamp examination was unremarkable in healthy individuals. In the study group, the mean OSDI value was 45.2 (range, 21.6 to 63.5). Eleven of them had meibomian gland dysfunction and four had aqueous deficient dry eye. Table 2 shows the summary estimates of all parameters evaluated in the study using various instruments.

Lipid layer thickness

The LLT measurements with IDRA and Lipiview did not show significant intraobserver difference in both control and DED patients (Table 3). The CoV was more in DED patients (IDRA, 26.96, Lipiview, 37.57) compared to control (IDRA,19.66, Lipiview, 36.85), although both groups had low repeatability. IDRA showed better repeatability compared to Lipiview. Bland-Altman analysis showed poor reproducibility and limits of agreement between two devices (First row of Figs. 1 and 2; Supplementary Table).

Table 3.

Intraobserver repeatability of tear film parameters for both Normal and DED patients analysed using Linear Mixed Effect modelling & Generalized Linear Hypothesis Test.

		NORMAL EYE					DRY EYE
		Mean Diff	LCL	UCL	P value	CoV	Mean Diff	LCL	UCL	P value	CoV
LLT	IDRA (Observer 1)	1.72	−0.29	3.72	0.09	19.66	4.3	−2.4	11	0.21	26.96
	Lipiview (Observer 1)	−1.67	−5.36	2.03	0.37	36.85	−4.03	−8.62	0.55	0.084	37.57
TMH	IDRA (Observer 1)	−0.03	−0.05	−0.01	0.005	31.71	0.02	−0.02	0.05	0.27	29.61
	Oculus (Observer 1)	0	−0.02	0.02	0.75	39.14	0.03	0	0.05	0.27	38.08
NIBUT	IDRA (Observer 1)	−0.19	−0.85	0.46	0.56	25.13	−0.51	−1.84	0.83	0.45	48.56
	Oculus (Observer 1)	0.25	−0.64	1.15	0.57	28.58	0.6	−0.6	1.8	0.33	47.78

CoV coefficient of variance, Diff difference, LCL lower class limit, UCL upper class limit.

Statistically significant p < 0.05 value are in bold.

Fig. 1. Scatterplots of Bland–Altman analyses in healthy controls.

x-axis represents the average of the measurements and y-axis the difference. Mean and the 95% limits of agreement (±2 SD) are shown as dashed lines parallel to x-axis.

Fig. 2. Scatterplots of Bland–Altman analyses in DED patients.

x-axis represents the average of the measurements and y-axis the difference. Mean and the 95% limits of agreement (±2 SD) are shown as dashed lines parallel to x-axis.

Tear meniscus height

The mean TMH measured with IDRA and K5M had high CoV in normal (IDRA, 31.7, K5M, 39.1) and DED eyes (IDRA, 29.61, K5M, 38.0) (Table 3 and Supplementary Table). The repeatability was poor for both the machines, though statistically insignificant for K5M (IDRA, P = 0.005). Bland-Altman analysis showed good reproducibility of both IDRA and K5M but poor agreement between them (Second row of Figs. 1 and 2, Supplementary Table.

Non-invasive tear break-up time

The average NIBUT values obtained using IDRA and K5M had lesser CoV in normal (IDRA, 25.13, K5M, 28.58) compared to DED eyes (IDRA, 48.6, K5M, 47.7) (Table 3). The intraobserver values did not show significant difference. Bland-Altman showed good reproducibility of both IDRA and K5M but poor agreement between them (Third row of Figs. 1 and 2). In normal group, fifteen eyes had a K5M NIBUT less than 10 s compared to 29 eyes when measured using IDRA. Between different diagnostic platforms, LLT, NIBUT and TMH were significantly different for same observer (Fig. 2; Supplementary Table).

Discussion

This study reports intra- and interobserver variability for TMH, LLT and NIBUT measurements for individual instruments as well as between different instruments. There were significant intraobserver differences noted in NIBUT measured using K5M or IDRA, LLT (IDRA), and K5M or IDRA TMH values. Between two observers, there were significant differences in LLT measurements obtained using Lipiview or IDRA. Between instruments, all the measurements (LLT, NIBUT and TMH) were significantly different for both observers. The variability was more in DED eyes which reflects heteroscedastic nature of DED. Also, the higher CoV either in control or DED eyes could be due to the variability in tear film itself or reflex tearing.

The clinical tests for dry eye such as Schirmer and fluorescein TBUT are considered unrepeatable due to subjective nature of measurements [11]. With the advent of automated measurements, majority of the tear film related data is now captured on these devices. However, there are different diagnostic platforms available in the market with varying results. The reported mean LLT values in DED patients are between 54 to 76 nm (SD,15–25) whereas 65 nm in healthy controls (33–100) overlaps with DED patients [12]. We found mean LLT values of 48.21 nm (range, 40–92 nm) and 58 nm (range, 32–99 nm) in control group using IDRA and Lipiview, respectively, which is similar to published studies. The intraclass coefficient of repeatability of LLT using Lipiview has been noted to be 16 nm and interclass 13 nm [7]. In our study, intraclass CoV was 36 nm in control and 37 in DED patients. The sample size in the study by Zhao et al. was 20 eyes compared to 60 eyes in the current study. This could be responsible for larger intraobserver variation; however, the difference was not statistically significant. Whereas IDRA machine showed CoV of 19 nm in control and 26 nm in DED subjects. No significant difference has been reported between LLT values obtained using IDRA versus Lipiview in one of the study, though CoV or limits of agreement were not evaluated as done in our study [13]. The available SD values were 24.29 nm and 13.01 nm for Lipiview II and IDRA, respectively in DED patients which are similar to our study (22 nm and 14 nm, respectively). Though good reproducibility was noted for both IDRA and Lipiview, their values were not comparable.

The average TMH value in healthy subjects using K5M has been reported as 0.27 ± 0.12 mm and 0.29 mm (range, 0.26–0.34 mm) with the intraclass correlation coefficients and coefficient of variation values of 0.914% and 16.4%, respectively for interobserver variation [6, 14]. For the intra-individual variation, the intraclass correlation coefficients and coefficient of variation values were 0.939% and 15.9%, respectively [14]. Their study did not use Bland-Altman analysis. The intraobserver CoV for K5M in the current study was high for both IDRA and K5M. The values did not show significant intra or interobserver variation for K5M. The differences in mean TMH using IDRA showed more intraobserver variation. Hence, K5M seems more reliable for measuring TMH values. IDRA ocular surface analyser was introduced as a comprehensive diagnostic system for tear film analysis that can measure automated LLT, TMH, NIBUT, blink rate, meibography, pupillometry and conjunctival hyperaemia grading. It had advantages over Lipiview as NIBUT and TMH could also be measured. The reported normal values of NIBUT, LLT and TMH using IDRA are 10.4 ± 2.4 s, 73.4 ± 21.9 nm, and 0.289 ± 0.16 mm, respectively [15]. However, repeatability values are not available for IDRA. We observed normal values to be 8.9 ± 0.34 s, 48.21 ± 1.49 nm and 0.21 ± 0.01 mm for NIBUT, LLT and TMH, respectively. The repeatability for TMH and LLT values was low, hence the same observer should be involved in performing dry eye measurements using the IDRA platform.

The variability in fluorescein TBUT is up to 8 s between two visits with ICC of 0.29 [11]. The automated measurements of NIBUT have shown better limits of agreement between visits compared to fluorescein TBUT [16]. NIBUT measured with the Keratograph (software version 2.73r19) in 100 healthy personnel ranged from 0.36 s to 29.0 s, with 63% of readings being <5 s and 85% <10 s [5]. The keratograph used in the current study was the latest one, 5 M. No healthy individual had value <5 s and only 25% had value <10 s. The reported average NIBUT using K5M in healthy eyes is 10.35 ± 4.2 s, 10.9 ± 3.9 s, which is comparable with our value of 11.9 ± 0.50 s [6, 8]. K5M showed fair repeatability for NIBUT but CoV is more in DED subjects compared to healthy controls. Both K5M and IDRA had good reproducibility but agreement is poor between the two. LLT cannot be measured as numeric value with K5M, hence we did not evaluate the lipid layer status using K5M. The limitation of this study was the number of eyes examined but 60 eyes were totally imaged thrice totalling 180 scans per patient and a total of 360 scans. Another limitation is unequal number of DED patients and normals with different proportions of sex and age between the groups. In Indian population, age and sex did not affect NIBUT values significantly, and a very weak effect of age was noted on TMH values [17]. The strength of this study is the evaluation of measurements across three dry eye diagnostic platforms, the results of which can be applied universally.

In conclusion, this study tried to assess the repeatability and reliability of different commercially available diagnostic platforms in the objective assessment of tear film parameters and if there exists any agreement between them. The study found that no two dry eye diagnostic platforms can be used interchangeably for the evaluation of tear film. The high coefficient of variation in DED patients reflects variability in tear film irrespective of the device used.

Summary

What was known before

• Oculus keratograph 5 M has shown variable (low or good) repeatability for NIBUT values and moderate reproducibility for TMH values.

• No significant difference has been reported between LLT values obtained using IDRA versus Lipiview.

• Lipiview shows good repeatability for LLT values.

What this study adds

• Three diagnostic platforms for tear film evaluation- IDRA, K5M & Lipiview cannot be used interchangeably.

• The high coefficient of variation in DED patients reflects variability in tear film irrespective of the device used.

• NIBUT using K5M & IDRA-OSA shows good repeatability and reproducibility in control group but poor repeatability in DED patients.

• TMH values obtained with K5M or IDRA-OSA had poor repeatability with high CoV.

Author contributions

SS and SB were responsible for designing the study protocol, writing the protocol and IRB, conducting the literature review and writing the manuscript along with interpreting results, and creating tables. SSA and ZM were responsible for performing the tear film imaging and extracting the data from machines and electronic records. SSA contributed to figure designing as well. MHA conducted the statistical analyses and contributed to analysing data and interpreting results.

Funding

Hyderabad Eye Research Foundation (HERF), India.

Data availability

Supplementary file contains the statistical data. Raw data are available with authors and can be shared upon reasonable request.

Competing interests

The authors declare no competing interests.

Supplementary information

The online version contains supplementary material available at 10.1038/s41433-022-02281-2.