Reproducibility and agreement in evaluating retinal nerve fibre layer thickness between Stratus and Spectralis OCT

Abstract

Purpose

To evaluate intra-device reproducibility of retinal nerve fibre layer (RNFL) measurements obtained using Stratus and Spectralis optical coherence tomography, and to analyze inter-device correlation and agreement for these measurements.

Design

Prospective observational study.

Methods

A total of 30 normal individuals participated in the study. One eye of each participant was scanned three times during one session by the same operator using Spectralis and Stratus. Intra-class correlation coefficients (ICCs), correlation coefficients (R), and Bland–Altman plots (BAPs) were used to assess reproducibility, correlation, and agreement between the two devices, respectively.

Results

A significant difference in mean RNFL thickness was seen between Stratus and Spectralis (106.2±6.9 μm vs100.0±7.3 μm, P=0.0001). ICCs of RNFL thickness measurements ranged from 0.69 (clock hour 2; 95% confidence interval (95% CI): 0.54, 0.85) to 0.91 (inferior quadrant; 95% CI: 0.86, 0.96) for Stratus and were higher for Spectralis, ranging from 0.87 (temporal-superior sector; 95% CI: 0.79, 0.94) to 0.96 (global and nasal-inferior sector; 95% CI: 0.94, 0.99). Rs of RNFL thickness measurements between the two instruments ranged from 0.61 (temporal quadrant) to 0.87 (superior quadrant). BAPs demonstrated a systematic difference in RNFL values between the two devices, with Spectralis producing thinner RNFL values than Stratus.

Conclusions

Spectralis demonstrated higher ICCs and thinner RNFL measurements than Stratus. Although the inter-device correlation was good, differences in RNFL measurements obtained by the two devices indicate that these measurements would not be interchangeable in clinical evaluations.

Introduction

Optical coherence tomography (OCT), first introduced by Huang et al¹ in 1991, is a non-contact, non-invasive imaging technology that uses low coherence interferometry to generate cross-sectional images of the retinal nerve fibre layer (RNFL), macula, and optic nerve head.^{2, 3, 4, 5, 6, 7} Two categories of OCT technology currently exist: time-domain (TD) OCT and spectral-domain (SD) OCT. TD OCT uses an interferometer to compare light reflection time from retinal structures with the reflection time from a reference mirror at a known distance. TD OCT analyzes one point at a time to obtain an A-scan and B-mode retinal image. In addition, it varies the length of the reference path with each acquired A-scan to determine depth information. These factors result in relatively low scanning speeds, susceptibility to motion artefacts, interpolation of RNFL thickness in areas between scans, and limited registration of sequential A-scans,^{2, 4} which may affect the reproducibility of the device. In contrast, SD OCT uses a spectrometer with Fourier transformation rather than the length of the reference path. It combines all light reflected from the retinal structures and the reference mirror to obtain depth information of all A-scans simultaneously. SD OCT acquires A-scans ∼100 times faster than TD OCT, which improves motion artefacts.^{8, 9, 10, 11}

A commercially available third-generation TD OCT (Stratus OCT; software version 4.0.1, Carl Zeiss Meditec, Dublin, CA, USA) provides image resolution of 8–10 μm (Stratus RNFL analysis printout is depicted in Figure 1a). Previous studies have demonstrated its good reproducibility of retinal measurements and high specificity and sensitivity for detection of glaucomatous damage.^{5, 6, 7} A commercially available first-generation SD OCT (Spectralis Heidelberg retinal angiography (HRA)+OCT; Heidelberg Engineering Inc., Heidelberg, Germany) provides optical image resolution of 7 μm and generates three-dimensional retinal images. Spectralis HRA+OCT consists of six modes: SD OCT, infrared imaging, fluorescein angiography, indocyanine green angiography, autofluorescence, and red-free photography. Recently, Spectralis HRA+OCT incorporated a new algorithm of RNFL analysis of the optic nerve head (Figure 1b).^{8, 9, 10, 11} As the new technology becomes more ubiquitous, patients with previous TD OCT imaging will likely undergo SD OCT imaging in the future. Therefore, correlation and agreement between the two technologies become clinically relevant to detect change in RNFL thickness in the long term.

Figure 1.

An example of retinal nerve fibre layer (RNFL) report obtained with Stratus OCT (a) and Spectralis HRA+OCT (b). Similar to Stratus OCT, Spectralis HRA+OCT depicts fundus images with a circular scan placed on the optic nerve heads, RNFL thickness profiles (black curve) along the circular scan plotted over the normative database values, RNFL measurements in μm corresponding to four quadrants, and inter-eye RNFL measurement asymmetry. In contrast to Stratus OCT, Spectralis HRA+OCT provides additional RNFL measurements in μm corresponding to six sectors and coloured bars indicating the overall classification (The name and birth date are fictitious. These data needed to be entered to obtain meaningful RNFL analysis.).

The purpose of this study was to evaluate the reproducibility of RNFL thicknesses obtained using Stratus and Spectralis. We analyzed correlation and agreement between the two devices among normal individuals to determine whether RNFL measurements obtained with these devices may be used interchangeably in clinical evaluations.

Materials and methods

All study participants were recruited between February and March 2009 from the employee pool in the eye clinic at the University of Minnesota. Informed consent was obtained from all participants following study approval by the Institutional Review Board of the University of Minnesota. The study adhered to the tenets of the Declaration of Helsinki. Subjects were included if they had no history of eye disease, best-corrected visual acuity of 20/25 or better, spherical refraction within ±6.0 D, and cylindrical refraction within ±3.0 D. Exclusion criteria included history of any eye surgery, clinical evidence of optic nerve or retinal pathology, cup-to-disc ratio of >0.5, and intraocular pressure of >22 mm Hg.

All participants underwent review of medical history, slit-lamp biomicroscopy, intraocular pressure measurement with tonopen (Tono-Pen XL, Reichert Ophthalmic Instruments, Depew, NY, USA), central corneal thickness measurements with pachymetry (DGH 55, Technology Inc., Exton, PA, USA), and fundus examination with a 78-D lens. Normal participants had healthy-appearing optic discs defined as the absence of glaucomatous optic neuropathy, focal or diffuse rim thinning, disc haemorrhages, or diffuse or localized RNFL defects.

One eye at the discretion of study participant was scanned for each participant. During one session, the same operator (SNA) scanned the selected eye three times using a Stratus device and then three times using a Spectralis device. Subjects leaned back and rested for 10 s between scans. If a scan did not meet quality criteria, one repeated attempt was made to obtain quality scan. All scans were obtained in dim lighting conditions to allow for physiological dilation of the pupil. The minimum pupil size was at least 5 mm. Internal fixation was used for all scans. Average, quadrant, clock hour, and sectoral RNFL measurements were obtained. The operator was masked to the results. All printouts were obtained after completion of the study.

Stratus OCT device

Stratus OCT, a third-generation TD OCT, is a non-contact imaging device that uses a light with wavelength of 820 nm to obtain real-time, cross-sectional tomographic images of the ocular structures. Details of this technique are described elsewhere.^{5, 6, 7} We used the fast RNFL thickness protocol, consisting of the three 3.45 mm diameter circle scan series compressed into one scan acquired in 1.92 s. Quality criteria included sharp scan beam and definition of vessels, scan beam centred on the optic disc, optic disc centred on the screen, even illumination, and signal strength ≥8.

Spectralis OCT device

Spectralis HRA+OCT, a first-generation SD OCT, is a non-contact imaging device that uses a light width wavelength of 870 nm to obtain real-time, cross-sectional tomographic images of the ocular structures. It combines OCT technology with a confocal scanning laser ophthalmoscope (Heidelberg Engineering Inc.). Details of this technique are described elsewhere.^{8, 9, 10, 11} We used the RNFL optic disc protocol, consisting of one 3.5–3.6 mm diameter circle scan (Spectralis HRA+OCT brochure; Heidelberg Engineering Inc., 2008). Quality criteria included sharp scan beam and definition of vessels, scan beam centred on optic disc, even illumination, automatic real-time (ART) score of 16, and signal-to-noise ration ≥15 dB.

Statistical analysis

Descriptive statistics were used to evaluate continuous and demographic data. Repeated-measures analysis of variance and intra-class correlation coefficients (ICCs) were used to assess the intra-device reproducibility of RNFL measurements. The ICC represents the ratio of the between-cluster variance to the total variance. It is thus an indicator of the proportion of variability attributable to the measurement itself, as opposed to the variation between different individuals in the parameter being measured. Values of the ICCs range from 0 to 1, where ‘0' indicates perfect disagreement and ‘1' indicates perfect agreement between repeated measurements.¹² The average of the three measurements obtained with each device was used to reduce the noise.

Correlation coefficients and Bland–Altman plots (BAPs)¹³ were used to assess the correlation and agreement between the two devices, respectively. Agreement is present when most of the data points are situated around the mean difference line and between the limits of agreement. The ‘limits of agreement' are defined as two times the SD of paired differences (d±2s, where d is the mean difference and s the SD of the differences).¹³ A P-value <0.05 was considered statistically significant. Statistical analysis was performed using Stata 8.0 statistical software (Stata Corp., College Station, TX, USA).

Results

A total of 36 individuals were initially enrolled in the study. Six individuals were excluded because at least one of their scans did not meet the scan quality criteria: two individuals did not get suitable measurements on both devices, three individuals—on Stratus, and one—on Spectralis. Of the remaining 30 individuals (25 women and 5 men), the mean age of study participants was 33.7±10.6 years (range, 20–60 years), mean IOP was 16.5±2.9 mm Hg (range, 13–22 mm Hg), and mean CCT was 554.17±40.11 μm (range, 499–639 μm).

The mean, SD, minimum and maximum values, and ICC with 95% confidence interval (95% CI) of RNFL measurements obtained using Stratus and Spectralis devices are summarized in Tables 1 and 2, respectively. In general, the Stratus device provided thicker RNFL measurements than the Spectralis device. A statistically significant difference between Stratus OCT and Spectralis HRA+OCT was found for mean RNFL thickness measurements (106.2±6.9 μm vs 100.0±7.3 μm, P=0.0001), superior RNFL thickness measurements (129.5±11.7 μm vs 121.0±13.5 μm, P=0.0001), inferior RNFL thickness measurements (138.3±13.0 μm vs 130.2±14.1 μm, P=0.0001), nasal RNFL thickness measurements (84.1±13.6 μm vs 73.8±12.6 μm, P=0.0001), but not temporal RNFL thickness measurements (73.1±12.5 μm vs 74.2±11.4 μm, P=0.1). The calculated cross-sectional area of RNFL represented by the average thickness multiplied by the scan circumference was 366.39 μm for Stratus and 350–360 μm for Spectralis.

Table 1. Mean±SD, minimum (min), maximum (max), and intra-class correlation coefficient (ICC) with 95% confidence interval (95% CI) of retinal nerve fibre layer (RNFL) thickness measurements obtained with Stratus OCT.

Region	Mean±SD (μm)	Min, max (μm)	ICC (95% CI)
Average	106.2±6.9	92.8, 122.3	0.86 (0.77, 0.94)
Superior	129.5±11.7	103, 154	0.83 (0.74, 0.93)
Inferior	138.3±13.0	115, 167	0.91 (0.86, 0.96)
Nasal	84.1±13.6	49, 113	0.78 (0.66, 0.90)
Temporal	73.1±12.5	54, 117	0.89 (0.82, 0.95)
Clock 1	121.6±16.9	87, 174	0.85 (0.76, 0.93)
Clock 2	99.3±14.3	59, 132	0.70 (0.54, 0.85)
Clock 3	69.6±16.6	39, 110	0.83 (0.74, 0.93)
Clock 4	83.3±16.6	48, 126	0.82 (0.72, 0.92)
Clock 5	120.8±16.5	91, 167	0.79 (0.68, 0.91)
Clock 6	152.9±18.7	118, 191	0.90 (0.84, 0.96)
Clock 7	141.3±22.6	93, 192	0.91 (0.85, 0.96)
Clock 8	74.6±13.8	51, 114	0.84 (0.75, 0.93)
Clock 9	56.6±9.2	41, 95	0.88 (0.80, 0.95)
Clock 10	88.2±17.5	64, 146	0.90 (0.84, 0.96)
Clock 11	137.9±18.4	98, 176	0.88 (0.81, 0.95)
Clock 12	129.0±20.6	92, 191	0.91 (0.85, 0.96)

Table 2. Mean±SD, minimum (min), maximum (max), and intra-class correlation coefficient (ICC) with 95% confidence interval (95% CI) of retinal nerve fibre layer (RNFL) thickness measurements obtained with Spectralis HRA+OCT.

Region	Mean±SD (μm)	Min, max (μm)	ICC (95% CI)
Global	100.0±7.3	87, 116	0.96 (0.94, 0.99)
Superior	121.0±13.5	98, 159	0.92 (0.88, 0.97)
Inferior	130.2±14.1	100, 161	0.92 (0.87, 0.97)
Nasal	73.8±12.6	42, 96	0.95 (0.93, 0.98)
Temporal	74.2±11.4	57, 115	0.94 (0.90, 0.98)
TS	133.9±12.8	103, 163	0.87 (0.79, 0.94)
NS	108.1±20.9	75, 184	0.94 (0.91, 0.98)
NI	115.8±19.3	76, 152	0.96 (0.94, 0.99)
TI	144.7±18.6	107, 192	0.90 (0.84, 0.96)

Sectors: TS, temporal-superior; NS, nasal-superior; NI, nasal-inferior; TI, temporal-inferior.

ICCs of RNFL thickness measurements ranged from 0.70 (clock hour 2; 95% CI: 0.54, 0.85) to 0.91 (inferior quadrant; 95% CI: 0.86, 0.96) for Stratus OCT (Table 1) and were higher for Spectralis HRA+OCT (Table 2), ranging from 0.87 (temporal-superior sector; 95% CI: 0.79, 0.94) to 0.96 (global and nasal-inferior sector; 95% CI: 0.94, 0.99). Comparable quadrant and average ICCs were lower for Stratus than for Spectralis (Tables 1 and 2). Correlation coefficients comparing RNFL thickness measurements between the two devices were above average and ranged from 0.61 (temporal quadrant) to 0.87 (superior quadrant) (Table 3).

Table 3. Correlation coefficients, mean difference (MD) with 95% confidence intervals (95% CIs), and limits of agreement (LoA) for retinal nerve fibre layer (RNFL) measurements of the optic nerve head obtained with Stratus OCT and Spectralis HRA+OCT.

Region	Correlation	MD (μm) (95% CI)	LoA (μm)
Average	0.73	6.2 (4.3, 8.1)	−4.1, 16.5
Superior	0.87	8.5 (6.1, 11)	−4.7, 21.8
Inferior	0.84	8.1 (5.3, 10.9)	−7.1, 23.3
Nasal	0.74	10.3 (6.9, 13.7)	−7.8, 28.4
Temporal	0.61	22.6 (19.3, 25.9)	4.7, 40.5

BAPs of the RNFL agreement between Stratus and Spectralis are depicted in Figure 2. Spectralis tended to produce lower average and quadrantic RNFL measurements than Stratus (Figure 2). The mean differences in RNFL measurements among quadrants ranged from 6.2 μm (average; 95% CI: 4.3, 8.1; limits of agreement: −4.1, 16.5) to 22.6 μm (temporal quadrant; 95% CI: 19.3, 25.9; limits of agreement: 4.7, 40.5) (Table 3).

Figure 2.

Bland–Altman plots of the retinal nerve fibre layer (RNFL) measurements obtained with Stratus OCT and Spectralis HRA+OCT. (a) Agreement for superior quadrant RNFL; (b) agreement for inferior quadrant RNFL; (c) agreement for nasal quadrant RNFL; (d) agreement for temporal quadrant RNFL; and (e) agreement for mean RNFL.

Discussion

Our study demonstrated ICCs and thinner RNFL measurements obtained with Spectralis than with Stratus. The overall correlation between the two devices was above average; however, the differences in RNFL measurements obtained by the two devices were large, indicating that these measurements would not be interchangeable in clinical evaluations. A difference of up to 5 μm would be small enough to be clinically acceptable.

Reproducibility of measurements is crucial for detection, evaluation of progression, and effectiveness of therapeutic management of the optic nerve disease. Stratus reproducibility for RNFL measurements has been extensively studied. Paunescu et al⁷ demonstrated reproducible RNFL measurements obtained with Stratus OCT 3 in healthy subjects with ICCs ranging from 0.53 for clock hour 6 RNFL to 0.83 for overall mean. Budenz et al⁵ studied average and quadrant RNFL reproducibility in 88 controls and 59 glaucoma patients using the standard and fast RNFL protocols, respectively. For the fast RNFL protocol, the authors found ICCs ranging from 0.84 for the nasal quadrant to 0.95 for average RNFL. A recent paper¹⁴ evaluated the reproducibility of fast and repeat RNFL protocols among 20 controls and 20 glaucoma patients. In their study, the ICCs for controls ranged from 0.74 for clock hour 1 RNFL to 0.93 for the average RNFL. Similar to other investigators, we found good reproducibility of Stratus. In our study, Stratus ICCs ranged from 0.70 to 0.91.

Various investigators have compared RNFL measurements using Stratus vs SD OCT, such as Cirrus high-definition (HD)-OCT,^{15, 16, 17, 18, 19} RTVue OCT,²⁰ and Topcon OCT.²¹ Knight et al¹⁵ noted thinner RNFL measurements produced by Cirrus and high correlation for mean RNFL measurements between the two devices (R²=0.95). Another study¹⁶ found (N=16) that Cirrus provided RNFL measurements that were highly reproducible (ICCs ranged from 0.81 for clock 1 RNFL to 0.98 for the temporal quadrant), thinner, and well correlated with Stratus RNFL measurements (R² ranged from 0.47 for the superior quadrant to 0.87 for average RNFL) among normal individuals. Kim et al¹⁷ demonstrated consistently thinner RNFL measurements and higher reproducibility obtained with Cirrus (ICCs ranged from 0.74 for clock 11 to 0.99 for clock 9 and the temporal quadrant) than with Stratus (ICCs ranged from 0.63 for clock 11 to 0.93 for clock 7) in 14 normal participants. Others have¹⁸ found better intra-visit and inter-visit variabilities for Cirrus HD-OCT than Stratus OCT, and inter-device proportional biases of RNFL measurements that increased with means. One paper noted higher sensitivity (63.6%) and specificity (100%) and thinner RNFL measurements obtained with Cirrus HD-OCT (85.6±14.6 μm) than with Stratus OCT (40, 96.7, and 98.0±18 μm, respectively). Some authors have²⁰ demonstrated highly reproducible (ICCs ranged from 0.97 for average RNFL to 0.91 for the superior and nasal quadrants) and thicker RNFL measurements obtained with RTVue in normals (N=30). The correlation between RTVue and Stratus was good (R² ranged from 0.44 for temporal RNFL to 0.81 for average RNFL). Hood et al²¹ did not find any significant systematic differences in RNFL measurements produced by Topcon and Stratus in their study, despite the fact that in their study, SD OCT tended to produce thicker RNFL measurements than TD OCT for glaucoma patients with thin RNFL. Although the results of our study cannot be compared directly with those that studied other OCT devices, similar to other investigators, we obtained thinner RNFL measurements with Spectralis than with Stratus among controls.

Most of the currently available data on Spectralis involve reproducibility or evaluation of macular thickness measurements rather than RNFL measurements. Studies have demonstrated higher reproducibility of macular measurements obtained with Spectralis than with Stratus and high correlation between the two devices.^{9, 22, 23} It is not surprising that Spectralis demonstrates greater reproducibility as the image resolution is higher, image acquisition is faster, coupled with its eye-tracking software, and ART and Heidelberg noise reduction functions to increase image quality.^{8, 9}

We found above average correlation between the Stratus OCT 3 and Spectralis HRA+OCT. However, a correlation simply indicates that a linear function between the two measurements exists, meaning that measurements obtained using one device increase as the increasing measurements obtained using the other device. BAPs produce a more in-depth analysis of relationship and differences between the measurements.¹³ In our study, the ‘limits of agreement' (two times the SD of paired differences) was between 8 and 40 μm (Table 3). Furthermore, we found a systematic difference in RNFL values between Spectralis and Stratus, with the latter consistently producing higher RNFL values than the former (Figure 2). Our findings indicate that RNFL measurements obtained using either Spectralis or Stratus among controls are not clinically comparable. Including glaucoma patients in the analysis would increase the range of values and the power of the comparison (both by ICC and by Bland–Altman analysis). It is possible that in glaucoma patients, RNFL measurements obtained using Stratus may be thinner than those obtained using Spectralis. Randomizing the order of OCT scanning would have helped to achieve a near balance of the number of times each OCT device came first.

The differences in RNFL measurements between the two devices may result from the following. Spectralis uses a larger diameter peri-papillary analysis ring than the Stratus for RNFL thickness. The RNFL becomes thinner with increasing distance from the edge of the optic disc, which may explain the systematically thinner measurements with Spectralis.²⁴ However, if the diameter of the scan circle was the only cause for the difference in RNFL measurements, the cross-sectional areas of the RNFL obtained with Stratus and Spectralis would be identical. As the cross-sectional areas are not identical (366.39 μm for Stratus and 350–360 μm for Spectralis), additional factors may account for differences in RNFL measurements. Stratus incorporates blood vessel thickness into the measurements, which can also result in thicker measurements.^{21, 25} Furthermore, it is reasonable to assume that both Stratus and Spectralis use different algorithms to determine RNFL thickness. Stratus localizes the top of the ganglion cell layer.¹⁵ At this time, Spectralis has not identified its RNFL algorithm for proprietary reasons.

As the ophthalmic community moves to SD OCTs on many different platforms using various software and glaucoma evaluation packages, the previous TD OCT measurements will require comparison with SD OCT to determine interchangeability. Our study demonstrated significant differences in RNFL measurements, suggesting that Spectralis OCT RNFL measurements are not comparable and clinically interchangeable with Stratus OCT. More studies are required to evaluate short- and long-term differences of RNFL measurements obtained using TD OCT and SD OCT among controls and patients.

Stella N Arthur, MD, MSPH: none; Scott D Smith, MD, MPH: none; Martha M Wright, MD: Allergan (C), Watermark Research Partners (C); Alana L Grajewski, MD: none; Qi Wang, MS: none; Joseph M. Terry, MD: none; and Michael S Lee, MD: Pfizer Ophthalmics (S), Teva Pharmaceutical Industries, Ltd (S), and Roche Laboratories (O).