Inter-grader Agreement of the Ocular Staining Score in the Sjögren’s International Clinical Collaborative Alliance (SICCA) registry

Abstract

Purpose

To determine the intra-observer and inter-observer reliability of a novel ocular staining score among trained ophthalmologists.

Design

Reliability analysis within a prospective, observational, multi-center cohort study.

Methods

Those enrolled in the National Institutes of Health-funded Sjögren’s International Collaborative Clinical Alliance (SICCA) who presented for follow up at the University of California San Francisco, Aravind Eye Hospital, Johns Hopkins University, and the University of Pennsylvania were included. Study participants were graded using the ocular staining score by at least two masked SICCA-trained ophthalmologists. The primary outcome for this study was the intraclass correlation coefficient (ICC) for the total ocular staining score. ICC’s were also calculated for tear break up time (TBUT), conjunctival and corneal staining.

Results

Total ocular staining score had an ICC of 0.91 for the right eye (95% CI 0.85 – 0.96) and 0.90 for the left eye (95% CI 0.83 – 0.97). Corneal staining (right eye 0.86, 95% CI 0.76 – 0.93, left eye 0.90, 95% CI 0.81 – 0.95) and conjunctival staining (right eye 0.87, 95% CI 0.80 –0.93, left eye 0.85, 95% CI 0.75 – 0.93) demonstrated excellent agreement. The ICC for TBUT was slightly lower (right eye 0.77, 95% CI 0.64 – 0.89; left eye 0.81, 95% CI 0.68 – 0.90).

Conclusions

Previous studies have shown that the ocular staining score is correlated with other diagnostic components of Sjögren syndrome. In this study, we demonstrate high reliability in grading among trained ophthalmologists, completing the validation of this test.

Introduction

Sjögren syndrome is a chronic autoimmune inflammatory disorder characterized by decreased exocrine function of the salivary and lacrimal glands.^{1, 2} It can be primary or secondary, when it is related to other autoimmune conditions such as systemic lupus erythematosus or rheumatoid arthritis.³ Due to diagnostic challenges and different diagnostic criteria, the exact prevalence of Sjögren syndrome has not been well established. One study, using the American European Consensus Group criteria for diagnosis, found an annual incidence of 4 per 100,000; 70% of these cases were primary.⁴ The American College of Rheumatology currently recommends that a classification of Sjögren syndrome be made based on meeting two of the following criteria: 1) positive serology for anti–SSA and/or anti-SSB; 2) ocular staining score of ≥ 3; 3) presence of focal lymphocytic sialadenitis with focus score ≥ 1 foci/4mm.^2,5

The Sjögren’s International Collaborative Clinical Alliance (SICCA) is an NIH-funded group of investigators who are exploring the etiology, diagnosis, epidemiology, and treatment of Sjögren syndrome in a large prospective cohort. In order to characterize keratoconjunctivitis (KCS) associated with Sjögren syndrome, a new quantitative ocular staining score was developed and tested in this cohort. In a prior study in this cohort, abnormal ocular staining score was strongly associated with other features of Sjögren syndrome.⁶ The reproducibility of dry eye measurements has been debated. In one study of a single grader who repeated Schirmer test, tear breakup time and cotton-thread test measurements on different days, the repeatability was poor.⁷

In this study, we evaluate the inter-grader and intra-grader reliability of the ocular staining score among SICCA-trained ophthalmologists examining the same patient on the same day. If the ocular staining score is shown to be reliable, it may serve as a useful diagnostic tool in the evaluation of patients with both Sjögren syndrome-related and non-Sjögren syndrome-related KCS.

Methods

Study participants who presented for their Sjögren’s International Collaborative Clinical Alliance (NIH N01 DE32636) study visit at University of California San Francisco, Aravind Eye Hospital, Johns Hopkins University, or University of Pennsylvania were included. The methods of the cohort study as well as the development of the ocular staining score, and standardized training of all involved ophthalmologists have been outlined previously.⁶ Five ophthalmologists participated in the reliability analysis. In accordance with the SICCA study, participants were asked not to apply their routine drops for the 12 hours preceding their visit and to discontinue contact lens wear for at least 7 days prior to examination. Two to three ophthalmologists scored each patient and were masked to each other’s ocular staining score grading. Institutional Review Board approval was obtained from all participating institutions for the Sjögren’s International Collaborative Clinical Alliance prospective cohort study, including this reliability analysis.

During the examination, study participants had tests and exam in the following order: un-anesthetized Shirmer test, slit lamp examination assessing tear break-up time (TBUT) with fluorescein, punctate epithelial erosions (PEEs) of the cornea with fluorescein and conjunctival staining patterns with lissamine green. Lid, conjunctival and corneal abnormalities were also noted. A subset of participants also had osmolarity using TearLab (TearLab Corporation, San Diego). Each parameter was graded sequentially by the different investigators who remained masked to each others’ grading.

Schirmer test I was performed prior to all other tests and before any drops were instilled in the eye. As soon as both Schirmer strips were in place the strips remained in place a maximum of five minutes or until completely saturated. Fluorescein (0.5% preservative free, Leiter’s pharmacy, San Jose, CA) was instilled immediately after removing the Schirmer strips. The cornea was examined at the slit lamp using 10x magnification and illumination set on “high” with the cobalt blue filter between 4 and 8 minutes after instillation. TBUT was defined as the time in seconds between the patient’s last blink and the first appearance of a dry spot on the corneal surface. It was measured times and the mean value was recorded. A value of 10 or greater was considered normal.⁸ Participants were then given a score of 0 if there were no PEEs, 1 if there were 1–5 PEEs, 2 if there were 6–30 PEEs and 3 if there were more than 30. They were then given an additional point each if they had patches of confluent staining, staining in the pupillary area, or one or more filaments for a maximum possible corneal staining score of 6.

After detailed assessment for lid, conjunctival and corneal abnormalities, lissamine green (1 drop of 1% lissamine green dye, Leiter’s pharmacy, San Jose, CA) was instilled in the eye. The conjunctival staining was then immediately assessed with 10x magnification with white illumination through a neutral density filter. Interpalpebral nasal and temporal conjunctiva were assigned a grade of 0 for no staining, 1 for 10–32 dots, 2 for 33–100 dots and 3 for greater than 100 dots. These values were then summed for a maximum possible score of 6 for conjunctival staining. The total ocular staining score was determined by adding the conjunctival and corneal staining with scores ranging from 0 to 12 for each eye. (Supplemental Figure).

Inter-rater agreement was assessed using the Intraclass Correlation Coefficient (ICC), estimated using analysis of variance methods for unbalanced data. Separate ICCs were estimated for the total ocular staining score, ocular staining score between the two eyes of an individual patient, and individual parts of the exam including TBUT, conjunctival lissamine staining pattern, and corneal fluorescein staining pattern. Precision of ICC estimates were summarized using percentile-based 95% bootstrap percentile confidence intervals based on resampling (n=999 resamples) drawn at the level of the patient to account for clustering of responses. Calculations were performed in Mathematica 10.0 (Wolfram Research, Champaign Illinois).

Results

Ninety-eight eyes of 49 study participants were evaluated and scored at University of California San Francisco, Aravind Eye Hospital, Johns Hopkins University, or University of Pennsylvania by at least two SICCA ophthalmologists between March 2011 and September 2012. One eye of one patient was excluded due to missing data. Table 1 outlines the characteristics of the study participants included. The majority of participants were female (n = 41, 84%) with a median age of 57 (IQR 51–67). Sixteen (33%) met criteria for a diagnosis of Sjögren syndrome using criteria defined by the American College of Rheumatology.⁵ Twenty-nine percent were positive for anti-Sjögren syndrome related antigen A (anti-SSA) or anti-Sjögren syndrome related antigen B (anti-SSB), while 20% were positive for rheumatoid factor (RF).

Table 1.

Baseline characteristics of study participants graded with Ocular Staining Score in the Sjogren’s International Collaborative Clinical Alliance (n = 49)

Characteristic	Number	%	Median (IQR)
Age, years			57 (51 to 67)
Gender, female	41	84
Serology positive for Anti-SSA/SSB	14	29
RF positivity	10	20
Focus Score ≥ 1	10	26
Meets ACR criteria for Sjögren syndrome	16	33
Shirmer’s mm/5min, right eye			9 (5 to 16)
Shirmer’s mm/5min, left eye			8 (5 to 18)
TBUT <10 seconds, both eyes	15	30	5 (4 to 7)
Ocular Staining Score (Abnormal ≥3), right eye			5 (2 to 8)
Ocular Staining Score (Abnormal ≥3), left eye			5 (3 to 8)

Abbreviations: IQR, interquartile range; RF, rheumatoid factor; ACR, American College of Rheumatology; TBUT, tear break up time;

Table 2 shows the ICCs for total ocular staining score, TBUT, conjunctival staining, and corneal staining. Total ocular staining score had an ICC of 0.91 for the right eye (95% CI 0.85 – 0.96) and 0.90 for the left eye (95% CI 0.83 – 0.97). The ICC between eyes for the same grader was 0.95 (95% CI 0.93 – 0.96). Cornea (right eye 0.86, 95% CI 0.76 – 0.93, left eye 0.90, 95% CI 0.81 – 0.95) and conjunctival (right eye 0.87, 95% CI 0.80 – 0.93, left eye 0.85, 95% CI 0.75 – 0.93) staining also demonstrated excellent agreement. ICC for TBUT was slightly lower (right eye 0.77, 95% CI 0.64 – 0.89, left eye 0.81, 95% CI 0.68 – 0.90), but improved when the TBUT in both eyes was averaged (0.81, 95% CI 0.68 – 0.91).

Table 2.

The Intraclass Correlation Coefficient for the Ocular Staining Score in the Sjogren’s International Collaborative Clinical Alliance

Exam element	Mean ICC (95% CI)
	Right	Left	Both Eyes
Total OSS score	0.91 (0.85 – 0.96)	0.90 (0.83 – 0.97)
Max OSS score			0.90 (0.83 – 0.96)
OSS score between eyes			0.95 (0.93 – 0.96)
Tear Break up Timea	0.77 (0.64 – 0.89)	0.81 (0.68 – 0.90)	0.81 (0.68 – 0.91)
Conjunctival stainingb	0.87 (0.80 – 0.93)	0.85 (0.75 – 0.93)
Corneal stainingc	0.86 (0.76 – 0.93)	0.90 (0.81 – 0.95)

Abbreviations: ICC, intraclass correlation coefficient; CI, confidence intervals; OSS, ocular staining score.

^aMeasured in seconds, average of three measurements. Both eyes represents average tear break up time between the two eyes.

^bCalculated by adding nasal and temporal staining

^cCalculated by adding corneal stain plus three bonus points for patches of confluent staining, staining in pupillary area, and presence of filaments.

Discussion

In this study, we demonstrate high intra and inter-grader agreement in ocular staining score among trained ophthalmologists. In a prior study, abnormal ocular staining score, defined as a score of 3 or greater, was associated with other findings of Sjögren syndrome such as a focal lymphocytic sialadenitis with a focus score of greater than 1, and positive serologic results for anti-SSA or anti-SSB antibodies.⁶ Although a score of 3 or greater was considered abnormal in that study, higher thresholds for an abnormal OSS may be more appropriate. Determining reliability is another important component of validating a new diagnostic test. Our study demonstrates that only a small amount of the variance in ocular staining score is due to the examiner. Prior studies of the repeatability of dry eye measurements typically repeated the test on different days, therefore it is unknown whether the lack of repeatability was due to intra- or inter-observer variation or to variability in the measurement over time.⁷ Although use of the ocular staining score for dry eye grading may improve reliability of dry eye measurements conducted on the same day, this study did not evaluate whether there are significant fluctuations in ocular staining score on different days. Given its high ICC, the ocular staining score has the potential to improve the internal validity of future dry eye studies as well as comparability between studies.

Agreement statistics have been calculated for determining the degree of clinical activity for other ocular diseases including trachoma,^{9, 10} as well as anterior^{11, 12} and posterior uveitis.¹³ There has been discussion about the magnitude of ICC or kappa that constitutes adequate agreement. Many things must be considered when interpreting agreement statistics, making it difficult to define an acceptable level of agreement for all tests. Recommendations have been published, for example, those by Landis and Koch, who described values varying from 0 – 0.20 representing “slight” agreement to 0.81 – 1 as “almost perfect” agreement for kappa.¹⁴ Although this serves as a rough guide, there were no data supporting these recommendations, and they are not widely accepted. Intraclass correlation coefficient estimates the proportion of the variance that is due to the patient and not the observer. Cohen clarifies that kappa is an approximation of ICC, and may at times be identical.¹⁵ Therefore, if these guidelines were accepted, the ocular staining score would have outstanding intra-observer and inter-observer reliability.

Individual portions of the exam including conjunctival and corneal staining pattern also demonstrated excellent agreement. However, a larger proportion of the variance in tear break-up time was due to the examiner than the variance of the total score. Taking the mean TBUT between the two eyes improved this marginally. Although there does not appear to be an advantage to using the maximum ocular staining score, it may be sufficient to calculate the score for only one eye since the ICC between two eyes is high. Only about one third of our patients met the American College of Rhematology criteria for a diagnosis of Sjögren syndrome. Twenty-eight percent of SICCA registry patients had abnormal ocular staining score without other evidence of Sjögren syndrome.⁶ These data suggest that the ocular staining score is appropriate to evaluate Sjögren and non-Sjögren syndrome KCS patients.

There are several limitations to our study. The patients available for repeated testing may not be representative of the entire SICCA registry or of KCS patients. However, the goal of this study was to measure agreement between examiners, therefore this is not likely to be an important consideration. The intensive training we performed in this study likely improves ICC, therefore these results may be most applicable to research environments or academic centers.

Previously the ocular staining score has been shown to correlate with other indices of Sjögren syndrome; here we complete the validation of the ocular staining score by demonstrating its repeatability among different examiners. The ocular staining score for dry eye grading may improve reliability of dry eye measurements, which is of particular value in clinical trials where this may be an outcome measure. These results also suggest that the ocular staining score may be useful in the diagnosis and treatment of patients with aqueous tear deficiency from causes other than Sjögren syndrome.

Supplementary Material

1. Supplemental Figure 1.

Sjögren’s International Collaborative Clinical Alliance ocular staining score form.

Biographies

Jennifer Rose-Nussbaumer

Dr. Rose-Nussbaumer specializes in corneal transplantation including lamellar keratoplasty and Boston keratoprosthesis. In addition to her clinical work, she is an NIH-funded clinical researcher with a current focus on corneal ulcer treatment in India and Nepal with the Proctor Foundation International Research Team. She is the co-principle investigator of Descemet Endothelial Thickness Comparison Trial, an ongoing surgical randomized controlled trial to evaluate outcomes of Ultrathin Descemet Stripping Endothelial Keratoplasty versus Descemet Membrane Endothelial Keratoplasty.

Dr. Bruce Gaynor, board certified in both internal medicine and ophthalmology, has broad experience in the treatment of infectious and inflammatory diseases of the eye. He has collaborated on the design, data collection and analysis of the ophthalmology portion of the Sjögren’s International Clinical Collaborative Alliance (SICCA) study since 2010.