Automated montaging and contrast stretching of adaptive optics ophthalmoscope images

Abstract

We present an automated montaging method for distortion-corrected single- and multi-channel adaptive optics ophthalmoscope images. The approach employs a zero-mean normalized weighted cross-correlation (ZNWCC) that enables assigning real-valued pixel weights to account for irregular image boundaries, spatially varying signal-to-noise ratio, and motion of retinal features. An initial sub-pixel pairwise registration using translation, rotation, and, if needed, scaling is refined with a global registration quality metric derived from peak ZNWCC values. The images are montaged through weighted averaging and contrast stretching that equalize mean image brightness and contrast in overlapping regions, compensating for differences in overall illumination, detector offset, and detector gain. A final montage is created by combining the transformed images calculated through a single interpolation to mitigate loss of resolution, and with an optional polynomial field-flattening step to increase contrast throughout the montage. The method is demonstrated on AO retinal image sets from human and mouse eyes.

1. Introduction

Adaptive optics (AO) ophthalmoscopy enables noninvasive visualization of the retina with subcellular resolution by imaging through larger, wavefront-corrected portions of the pupil than those used by conventional ophthalmoscopes. This imaging technology has been used to study human ocular [1–7] and neurodegenerative conditions [8–10], as well as the visual system and models of eye disease in non-human primates [11–14], pigs [15,16], ground squirrels [17,18], tree shrews [19], rats [20,21], and mice [22–29]. The superior transverse resolution of AO ophthalmoscopes comes at the expense of a reduced isoplanatic patch [30]. Consequently, large retinal areas can only be characterized by sequentially capturing images with small and partially overlapping fields of view for posterior montaging, a process also known as tiling, stitching, or mosaicking.

The need for montaging arises across a broad range of disciplines, including astronomy [31], radar [32], remote sensing [33], photography [34], microscopy [35–46], and biomedical imaging [47]. Conceptually, the montaging of distortion-corrected images comprises one or more of the following steps: establish pairwise image connections when relative positions are unknown or unreliable; perform pairwise image registration; refine registration through a global optimization process; and blend images across overlapping areas.

The establishment of pairwise image connection and registration is commonly achieved through feature-based or region-based registration algorithms. In the former, features common to each image pair are identified, matched, and used to estimate a geometric transformation that maps one image onto the other [39,40,46]. In practice, a predetermined minimum number of common features is required to assert that two images are connected (i.e., overlap), providing a robust threshold for reliable image registration. Region-based registration operates by exploring geometric transformation spaces seeking to maximize similarity metrics between image pairs, such as phase correlation [36,37,40,43], normalized cross-correlation (NCC) [39], and zero-mean normalized cross-correlation (ZNCC) [35,37,38]. These algorithms are appealing because they are easy to use and troubleshoot. Feature- and region-based registration methods are sometimes used in combination [36–39], and have been employed to estimate image translation [35–40,43–46], rotation, scaling, and a subset of affine transformations [39,40,43]. A montage can be refined by applying small geometrical transformations to each image, seeking to optimize a global registration metric rather than individual pairwise registration metrics.

The registered images are montaged with blending of overlapping regions, seeking to produce seamless transitions between images. Blended pixel values may be taken from a single image [48], calculated as a maximum intensity projection [39], or be the result of combining overlapping and/or adjacent pixels [35–37,41–43].

Feature-based montaging of AO retinal images has been demonstrated using scale-invariant feature transform (SIFT) algorithms [48–52], sometimes followed by removal of spurious matching of features by random sample consensus (RANSAC) and other methods [48,50]. Region-based registration has also been demonstrated using phase correlation [52], cross-correlation [53], and zero-mean normalized cross-correlation [54], by applying translation [51,54], translation and rotation [48,54], translation, rotation, and scaling [52], and affine transformations [50]. Blending of AO retinal images has been shown by selecting pixel values from one of the overlapping images [48], averaging of overlapping pixel values [52], applying a linear transition between images [50,51], and cropping along an irregular boundary [54].

Registration of AO retinal images must be robust to changes in feature intensity due to eye motion, such as those seen at the bottom of the foveal pit (foveal reflex; see Fig. 1) or Gunn’s dots [55]. Some retinal features exhibit pronounced intensity fluctuations due to physiological processes, including the photoreceptor outer segment mosaic [56–59] and the nerve fiber layer puncta [60]. Other retinal features, such as hyalocytes and blood cells [25,61–64] move.

Fig. 1.

Reflectance confocal adaptive optics scanning light ophthalmoscope images of the foveal reflex (bottom of foveal fit), showing substantial intensity variations due to small involuntary eye movement, posing a registration challenge (see Visualization 1 ^{(6.7MB, avi)} ).

In what follows, we propose and demonstrate a sequence of steps for montaging AO retinal images in the absence of vignetting, using physically plausible geometric and intensity transformations. The approach, summarized in the flow diagram in Fig. 2, relies on a recently proposed zero-mean weighted normalized cross-correlation (ZNWCC) [65] in which real-valued weight masks can be used to encode image boundaries, signal-to-noise ratio (SNR), the exclusion of pixels from the registration to improve robustness to feature motion or brightness changes, or combinations of these factors. An initial montage is created using pairwise registration of images and partial montages created through weighted averaging and contrast stretching to equalize mean image brightness and contrast across overlapping regions. The montage is then refined through optimization of a global ZNWCC-derived registration quality metric and a single interpolation of the original montage tiles to minimize interpolation blur, before an optional field flattening using a low-order two-dimensional polynomial fitting of the montage pixel values.

Fig. 2.

Flow diagram for montaging adaptive optics retinal images with steps required for all ophthalmoscope types, additional steps specific to scanning ophthalmoscopes and optional steps.

2. Image pre-processing

2.1. Distortion correction

Correcting image distortions introduced by the ophthalmoscope and by eye movements is a required preliminary step in montaging. Image distortion due to the optics of the eye, however, is not necessary, provided the relative position of the ophthalmoscope pupil to the eye does not change during the capture of all images to be montaged.

Raw images from all AO ophthalmoscopes exhibit optical distortions that can vary with refractive (focus) error and, when available, field of view steering [66–68]. These distortions can be measured with a model eye [69,70]. In scanning AO ophthalmoscopes, variations in scanning velocity and/or imperfect scan orientation introduce additional image distortions. Finally, resonant scanners that are not perfectly synchronized with the ophthalmoscope’s pixel acquisition introduce sampling jitter, which can be measured by recording the scanner’s analog orientation or velocity signals [71,72]. Eye motion during image capture in scanning ophthalmoscopes also induces image distortions, which can be corrected with various scanning and image processing techniques, including orthogonal scanning [73,74], Lissajous scanning [75], dual imaging [76], and strabismic imaging [77], or mitigated by inference from a sequence of images [78,79].

Distortion-corrected AO retinal images often have irregular boundaries and are cropped to a smaller, inscribed rectangular region of interest before montaging [54]. Here, we propose storing the distortion-corrected images as two equal sized and aligned rectangular arrays: a zero-padded array that fully contains the image, and a binary mask indicating which pixels lie inside its boundary, as illustrated in Fig. 3. As we shall see later, this binary array can be used as a weight mask in the ZNWCC registration to avoid discarding pixels [65].

Fig. 3.

Reflectance confocal (top row) and offset pinhole (middle row) adaptive optics scanning light ophthalmoscope images of a human photoreceptor mosaic before (left) and after (right) correction of optical, scanning and eye movement distortion. The binary mask (bottom row) indicates which pixels are inside the distortion-corrected images (white).

In addition to distortion, montaging should account for potential scaling variations across images in eyes with defocus, because the retinal area within the field of view changes with the distance between the eye and the ophthalmoscope (see Fig. 8 in Ref. [80]). Unfortunately, such scaling variations between images intended for montaging are rarely known, and thus this scaling must be estimated during the montaging process itself.

2.2. Registration of image sequences to create tiles and weight masks

Most AO ophthalmoscopes acquire sequences of short exposure images at each retinal location of interest using low retinal irradiance (energy per unit time per unit area) to reduce the risk of thermal light damage [81], as well as motion blur and, in scanning ophthalmoscopes, motion distortion. Here we co-register the images within each sequence using the ZNWCC and the binary arrays mentioned above as weight masks, before averaging to improve SNR. Here we refer to these distortion-corrected registered average images as tiles, which will be used for montaging.

If the ophthalmoscope has multiple synchronous channels with identical magnification, fixed spatial offsets and the same illumination wavelength, image registration can benefit from using all channels simultaneously in the ZNWCC. Multi-wavelength images, however, show wavelength inter-channel shifts, scaling and distortion [82–84] due to ocular longitudinal and transverse chromatic aberrations, which vary across the retina and with pupil movement. Therefore, only channels collecting the same wavelength of light should be used for co-registration of image sequences.

Regardless of weighting, image registration using a normalized cross-correlation function (NCC, ZNCC, NWCC or ZNWCC) consists in finding the coordinates of the function maximum within a region or interest that often corresponds to a minimum acceptable overlap. Rotation and scaling parameters can be estimated by repeating this procedure on rotated and scaled versions of one of the two images. Importantly, registration of AO retinal images using normalized cross-correlation functions, such as ZNWCC, is broadly applicable and has been demonstrated in both coherent and incoherent imaging modalities [85–87].

Once the relative positions of the N transformed (shifted, rotated and/or scaled) images $I_n^{\prime }$ and their binary masks $M_n^{\prime }$ have been estimated, a sum map can be defined as,

$$S(i,j,k)={\sum }_{n=1}^N{M}_n^{\prime }(i,j,k),$$ (1)

with the indices i, j and k denoting pixel row, pixel column and image channel, respectively. This sum map can be used to calculate montage tiles as,

$$I_{\text{avg}}(i,j,k)=\frac{{\sum }_{n=1}^N{M}_n^{\prime }(i,j,k)I_n^{\prime }(i,j,k)}{S(i,j,k)}.$$ (2)

The inclusion of the transformed masks in the numerator ensures that the domain of the transformed images to average are well defined when implementing this formula computationally. Similarly, a standard deviation map for each tile can be defined as,

$$\sigma (i,j,k)=\sqrt{\frac{{\sum }_{n=1}^N{[M_n^{\prime }(i,j,k)I_n^{\prime }(i,j,k)-I_{\text{avg}}(i,j,k)]}^2}{S(i,j,k)}}$$ (3)

Figure 4 shows an example tile and the corresponding sum and standard deviation maps for a sequence of photoreceptor mosaic images. The sequence was registered first with rigid translations (left column) and then with translations plus rotations (right column), illustrating the importance of correction of cyclotorsion to mitigate image blur.

Fig. 4.

Sum maps, registered averages and standard deviation of a sequence of distortion-corrected photoreceptor mosaic images, captured with a confocal adaptive optics scanning light ophthalmoscope. Left column panels correspond to registration using translation only, with noticeable blur due to cyclotorsion, while the right column corresponds to registration using translation and rotation, showing less blur and lower standard deviation values (14 vs. 20%).

2.3. Weight masks for montaging

Sum and standard deviation maps can be used to derive single- or multi-channel weight masks to improve pairwise registration of tiles with the ZNWCC [65], as illustrated below.

A valid binary weight mask, such as that originally proposed by Padfield [88] can be used to select tile pixels that result from averaging a minimum of $N_{\text{a}}$ overlapping images in a sequence, as a means to ensure a minimum SNR,

$$W_{\text{valid}}(i,j,k)=[S(i,j,k)>N_{\text{a}}].$$ (4)

Notably, the valid mask regions do not have to be convex or even connected. More generally, an SNR mask can be defined as

$$W_{\text{SNR}}(i,j,k)=W_{\text{valid}}(i,j,k)f(\text{SNR}(i,j,k)),$$ (5)

with f being any desired monotonically increasing function. Importantly, both f and SNR can be different across ophthalmoscope imaging channels. In a Poisson (i.e., photon) noise limited scenario, we could define an SNR mask as,

$$W_{\text{SNR}-\text{PN}}(i,j,k)=W_{\text{valid}}(i,j,k)\sqrt{S(i,j,k)I_{\text{avg}}(i,j,k)}.$$ (6)

Alternatively, weight masks can also be used to reduce or suppress the contribution of tile pixels whose intensity changes over time. One such mask can be defined as

$$W{(i,j,k)}_{\text{motion}}=W_{\text{valid}}(i,j,k)\left[1-\frac{\sigma (i,j,k)}{I_{\text{avg}}(i,j,k)}\right],$$ (7)

with examples showing intensity fluctuations and feature movement (e.g., blood cells) can be seen in Fig. 5 and Fig. 6, respectively. Weight masks can also be generated by combining criteria; for example, by multiplying the masks produced by each criterion.

Fig. 5.

Sum maps, registered averages, standard deviation and motion weight mask (see Eq. (7)) of a sequence of distortion-corrected photoreceptor mosaic AOSLO images with reflectance confocal (left) and offset pinhole (right) channels. Pixels outside the registered average image boundary (white) are computationally represented as not-a-number (scale bar 0.1°).

Fig. 6.

Sum maps, registered averages, standard deviation and motion weight mask (see Eq. (7)) of a sequence of distortion-corrected blood vessel AOSLO images with reflectance confocal (left) and offset pinhole (right) channels. Pixels outside the registered average image boundary (white) are computationally represented as not-a-number (scale bar 0.1°).

Mask selection should be guided by the retina layer being imaged, individual image SNR, and imaging modality. Applying these masks is not intended to enhance fine-grained registration accuracy; instead, their use aims to reduce the likelihood of large mis-registration by suppressing or attenuating spurious peaks of the normalized cross-correlation function used for registration.

3. Contrast matching

Brightness and contrast can vary substantially between tiles for several reasons, including head or eye movement in relation to the ophthalmoscope (e.g., due to vignetting or Stiles-Crawford imaging effect), detector gain and offset changes, light source power changes, eyelid position changes, and tear film breakup. Consequently, adjusting contrast is necessary for achieving seamless transition between partially overlapping tiles.

Let us assume that, in the absence of noise and vignetting, the intensity of distortion-corrected tiles $I_n^{\prime }(i,j,k)$ and $I_m^{\prime }(i,j,k)$ over the overlapping area differs only by a multiplicative factor $g_{n,m}$ (gain) and an additive constant $o_{n,m}$ (offset),

$$\begin{array}{ll}& W_{\text{valid},n}(i,j,k)W_{\text{valid},m}(i,j,k)I_n^{\prime }(i,j,k)=\\ & W_{\text{valid},n}(i,j,k)W_{\text{valid},m}(i,j,k)[g_{n,m}(k)I_m^{\prime }(i,j,k)+o_{n,m}(k)]\end{array},$$ (8)

where $W_{\text{valid},n}(i,j,k)$ and $W_{\text{valid},m}(i,j,k)$ are the binary masks calculated using Eq. (4) for tiles n and m, respectively, included here to ensure that the tiles are well defined when implementing this formula computationally. The gain and offset can be estimated using the tiles average pixel values ( ${\mu }_n$ and ${\mu }_m$ ) and their standard deviations ( ${\sigma }_n$ and ${\sigma }_m$ ) over their valid overlapping areas, which substituting in Eq. (8) gives

$$\begin{array}{ll}& {\mu }_n(k)=g_{n,m}(k){\mu }_m(k)+o_{n,m}(k)\\ & {\sigma }_n(k)=g_{n,m}(k){\sigma }_m(k)\end{array},$$ (9)

Solving for $g_{n,m}$ and $o_{n,m}$ yields,

$$\begin{array}{ll}& g_{n,m}(k)\approx \frac{{\sigma }_n(k)}{{\sigma }_m(k)},\\ & o_{n,m}(k)\approx {\mu }_n(k)-\left[\frac{{\sigma }_n(k)}{{\sigma }_m(k)}\right]{\mu }_m(k),\end{array}$$ (10)

which can be used to define the contrast-stretched tiles as

$$I_{m,\text{CS}}^{\prime }(i,j,k)=g_{n,m}(k)I_m^{\prime }(i,j,k)+o_{n,m}(k),$$ (11)

and can be used to calculate the pairwise montage as their weighted average,

$$I_{\text{montage}}(i,j,k)=\frac{W_n(i,j,k)I_n^{\prime }(i,j,k)+W_m(i,j,k)I_{m,CS}^{\prime }(i,j,k)}{W_n(i,j,k)+W_m(i,j,k)},$$ (12)

where the $W_n$ and $W_m$ can be any of the weight masks discussed previously. As an illustrative example of this approach, a pair of tiles in Fig. 7 are shown montaged using the sum maps as average weight functions, with and without contrast stretching.

Fig. 7.

Pairwise registration of AOSLO confocal reflectance foveal images, before and after contrast matching, using their frame sum maps as weighting masks for averaging. The yellow outline corresponds to valid (2^nd row) and overlap (3^rd row) areas.

4. Montaging

4.1. Strategy

The strategy proposed next proceeds iteratively through the set of images to be montaged: at each step, the pair of images with the highest ZNWCC peak are registered and merged after contrast stretching, the pair is replaced in the set by the merged image (i.e., a partial montage), and this process is repeated until a single montage remains or no remaining image or partial montage pair meets a user-defined minimum acceptable peak ZNWCC threshold.

The pairwise (single- or multi-channel) registration is achieved by generating rotated and/or scaled versions of one tile and comparing them against the other tile using the maximum ZNWCC value that corresponds to a minimum acceptable overlap. The peak ZNWCC value that corresponds to the optimal translation, rotation and/or scaling, is placed in a triangular matrix, as shown on the top of Fig. 8 for an initial set of six tiles. When the relative positions of the images are known, fewer matrix entries are required, reducing both computational cost and the likelihood of erroneous pairing. After the matrix is populated, we identify the pair of tiles corresponding to the maximum matrix element, merge them, and replace the pair of tiles with this partial montage. Then, we remove the rows and columns for the tile pair and compute a new row and column for the partial montage and repeat, as shown in Fig. 8, Fig. 9, and Fig. 10, producing partial montages with equal or greater overlap on each iteration, increasing confidence in their correspondence.

Fig. 8.

Montaging iterations 1 and 2 applied to a set of 6 original images with their sum maps (yellow trace indicates valid region), which are used as averaging weight masks, and matrix with maximum ZNWCC values between images assuming a minimum acceptable overlap.

Fig. 9.

Montaging iterations 3 and 4 applied to a set of 6 original images with their sum maps (yellow trace indicates valid region), which are used as averaging weight masks, and matrix with maximum ZNWCC values between images assuming a minimum acceptable overlap.

Fig. 10.

Final montage of a set of 6 original images and the sum of sum maps (yellow trace indicates valid montage region). See full resolution montages and sum maps in Visualization 2 ^{(12.2MB, tiff)} and Visualization 3 ^{(12.2MB, tiff)} , respectively.

4.2. Global registration metric

A montage quality metric can guide global optimization strategies, enable comparisons among montaging algorithms, and inform the selection of montaging algorithm parameters. These quality metrics can be generated as a combination of pairwise metrics, such as average intensity difference [54], normalized cross-correlation [48] and normalized mutual information [48,52]. Here, we propose the sum of the ratio of the peak ZNWCC value from the actual registration to the peak ZNWCC value from the best pairwise registration as a metric, that is,

$$Q_{\text{montage}}=\frac{1}{N}{\sum }_n^{N_{\text{pairs}}}\frac{\text{ZNWCC}(n)}{\text{ZNWC}{\text{C}}_{\text{max}}(n)},$$ (13)

where the index n runs through the $N_{\text{pairs}}$ possible tile pairs with minimum acceptable overlap. This metric yields values in the [0, 1] interval, with higher values indicating superior registration. However, for montages in which each new tile overlaps only with one other tile, the ratio remains 1.0, providing no information about registration quality. The metric can take values below one only when a new tile with minimum acceptable overlap with two or more montaged images is added to the montage (see example in Fig. 11). This might appear as a limitation of this particular metric, but in fact, it is a fundamental registration problem that no metric can address.

Fig. 11.

Outlines of an imaginary montage in which images were added in their numerical order, and only the fourth one has substantial (more than a corner) overlap with more than one other image when added to the montage.

4.3. Global optimization

Pairwise registration errors can accumulate as tiles are added to a montage. When tiles form a one-dimensional chain (for example, a single row or column), misalignment can be substantial yet not readily apparent, since each new tile overlaps only with its neighboring tiles on either side. Therefore, we implemented a refinement step, in which the montage is recreated by adding one tile at a time, with their position and rotation varied with fine (sub-pixel) steps to increase the montage quality metric, rather than the pairwise registration metric. Because this is done with only one tile at a time, it is not guaranteed convergence to the quality metric maximum possible value. In fact, different tile transformations will be found depending on the tile recall order. The approach, however, guarantees that the metric value does not decrease, and can be repeated multiple times, potentially with different recall orders.

The output of this approach is demonstrated by applying a single refinement iteration with the montage of AOSLO photoreceptor mosaic tiles shown in Fig. 12, together with the corresponding tile translation vectors and rotation angles, both with respect to the tile’s top-left corner. Importantly, the magnitude of the translation vectors corresponds to a combination of rotation and translation.

Fig. 12.

Montage of AOSLO photoreceptor mosaic tiles created using pairwise registration and refined using a (single iteration) global optimization, and the corresponding sum map montage with (×20) yellow arrows showing the translation of the top left corner pixel of each tile. The yellow circle shows the corner of the tile used as the optimization starting point, which is not translated or rotated. The plots show the translation vectors and rotation angles.

4.4. Re-sampling

The proposed iterative addition of tiles to the partial montage(s) requires the generation of shifted, rotated and/or scaled tiles through resampling of the original tiles at new pixel coordinates. These new pixel values are computed by interpolation, in this case bilinear because of its low computational cost. Some portions of the montage may be resampled multiple times, resulting in blur that can be substantial (as illustrated in Fig. 13). To mitigate this interpolation blur, once montaging is complete we register each original tile to the montage with blur and create a single transformed version for each tile using just one interpolation. The singly interpolated tiles, this time using bicubic interpolation for increased fidelity, are then combined through weighted averaging and contrast stretching to form a new montage with superior contrast and resolution, as illustrated in Fig. 13.

Fig. 13.

Montage of AOSLO photoreceptor mosaic tiles created through multiple interpolations during pairwise registration (top) and through a single interpolation (bottom), with the latter showing superior resolution.

4.5. Field flattening

Montage contrast can vary due to factors such as changes in retinal reflectivity, underlying retinal pathology, vignetting from misalignment of the eye and ophthalmoscope optics, and nonuniform retinal illumination. Therefore, and just for cosmetic purposes, the montage can be field flattened through division or subtraction of a low-order polynomial fitting, such as those shown in Fig. 14. The order of the polynomial can be selected for each montage based on personal preference.

Fig. 14.

Montage of AOSLO photoreceptor tiles field flattened through by low-order polynomial fits of the intensity (left), through division (middle) or subtraction (right).

5. Imaging instrument

Two custom AO scanning light ophthalmoscopes (AOSLOs), one for human subjects and one for small animals, were used to capture sequences of reflectance images with partially overlapping fields of view. The human data was collected using strabismic imaging to estimate and correct eye movement [77]. Both instruments create an imaging raster using a resonant optical scanner (13.8 KHz) and a non-resonant scanner operating at 14.9 Hz [29,89,90]. The AO consisted of a protected silver coated deformable mirror with 97 actuators (Alpao SAS, Montbonnot, France) and custom Shack-Hartmann wavefront sensor (SHWS). Images from confocal photomultipliers (PMTs) with sub-Airy pinholes [91], non-confocal quadrant PMTs [92] and the resonant scanner orientation signal (to correct sinusoidal warping and sampling jitter [71,72]) were captured simultaneously using a 16-channel digitizer operating at 40 MHz sampling rate (ATS9416 by Alazar Tech. Inc., Pointe-Claire, QC, Canada).

6. Imaging

6.1. Human subjects

Three subjects with no known ocular pathology were recruited for this study. Research procedures and informed consent followed the tenets of the Declaration of Helsinki and were approved by the institutional review board of Stanford University. One drop of 2.5% phenylephrine and one drop of 1% tropicamide were used to dilate the pupil and induce cycloplegia before retinal imaging. A bite-bar with a soft dental impression was attached to a manually operated three-axis translation stage to align and stabilize the subject during imaging.

For subject ADS_000651, sequences of 100 raw images with a 1.0 deg square field of view were captured at retinal locations with 0.25 deg overlap. For subject ADS_000669, sequences of 80 raw images with a 1.0 deg square field of view were captured at retinal locations with 0.35 deg overlap, showing the foveal avascular zone, and sequences of 150 raw images showing the photoreceptor mosaic. Each tile for montaging by averaging the 10 images with the highest peak ZNWCC between the strabismic channels after distortion correction; these images were rigidly registered (translation-only) and averaged. For two raw image sequences at adjacent retinal locations in subject ADS_000651, the number of averaged images was increased to 25.

6.2. Animal preparation

C57BL/6J mice (Jackson Labatories, Bar Harbor, Maine) were anesthetized by xylazine and ketamine based on their body weight (0.01 mg xylazine/g + 0.08 mg ketamine/g). Eye pupils were dilated and cycloplegia induced with one drop of 1% tropicamide (Somerset Pharma LLC, NJ). A zero diopter rigid PMMA clear contact lens with 1.6 mm base curvature and 3 mm diameter (Advanced Vision Technologies, Lakewood, Colorado) was used for corneal hydration. Ophthalmic ointment was used for hydration of the non-imaged eye and for brushing the whiskers away from the cornea (GenTeal Tears Ointment (Alcon, Fort Worth, TX) or Neomycin and Polymyxin B Sulfates and Bacitracin Zinc Ophthalmic Ointment, USP (Bausch & Lomb, Bridgewater, NJ). All procedures were performed in compliance with protocols approved by the Institutional Animal Care and Use Committee at Stanford University.

At each retinal location, sequences of 80 raw images were captured with 3.0 deg field of view (1.0 deg vertical overlap, 0.5 deg horizontal overlap); from these, the 20 images with the highest peak ZNWCC between the strabismic channels after distortion correction were rigidly registered (translation-only) and averaged to form montages.

7. Results

Three montages are presented next with magnified areas of tile overlap to illustrate the seamless transition between tiles. The registration of the distortion-corrected image sequences to create tiles for montaging was performed considering translations and rotations but not scaling, because the distance between the instrument and the eye was kept constant during imaging.

The first example, shown in Fig. 15, is a montage of 41 confocal and split-detection tiles of a human photoreceptor mosaic, together with their corresponding sum map. This montage was created using only the confocal images for registration and division using a third order polynomial intensity fitting for field flattening after global optimization that improved the quality metric from 0.911 to 0.941. Despite this field flattening, substantial contrast variation across the montage remains, as can be seen by comparing the highlighted regions of interest on the left and right sides of the montage. Also, it should be noted that because the tiles are predominantly arranged along a horizontal band, it is difficult to appreciate any potential accumulation of small pairwise registration errors, as the highlighted regions of interest illustrate (i.e., the tile boundaries are not obvious).

Fig. 15.

Montage of AOSLO reflectance tiles of a human photoreceptor mosaic: sum maps, confocal channel, non-confocal split-detection channel, quality metric 0.941 (scale bar: 1°, subject ADS_000669). See full resolution montages and sum maps in Visualization 4 ^{(87.6MB, tiff)} , Visualization 5 ^{(87.6MB, tiff)} and Visualization 6 ^{(87.6MB, tiff)} , respectively.

The second example, shown in Fig. 16, is a montage of 29 tiles of a human photoreceptor mosaic, in which most images overlap with three or more adjacent tiles, making it easier to reveal the potential accumulation of registration errors. In this montage, a single iteration of the global optimization improved the quality metric from 0.944 to 0.953.

Fig. 16.

Montage of AOSLO reflectance tiles of a human photoreceptor mosaic: sum maps, confocal channel, non-confocal split-detection channel, quality metric 0.953 (subject ADS_000651). See full resolution montages and sum maps in Visualization 7 ^{(32.7MB, tiff)} , Visualization 8 ^{(32.7MB, tiff)} and Visualization 9 ^{(32.7MB, tiff)} , respectively.

The final example in Fig. 17, shows a montage of 9 tiles showing the nerve fiber layer in a mouse eye without field flattening. The montage was assembled using registration weight masks generated as the standard deviation of the set of distortion-corrected images averaged to increase SNR before montaging. The use of these masks mitigated potential registration degradation due to the movement of blood cells within the field of view. In this montage, a single iteration of the global optimization improved the quality metric from 0.986 to 0.992. The montaging of the frame sum maps shows that eye movements in the anesthetized mouse were minimal, requiring only a small overlap between adjacent tiles, which was achieved with a motorized stage [93].

Fig. 17.

Montage of AOSLO reflectance sum map and tiles showing the nerve fiber layer in a mouse eye (quality metric 0.992), and magnified regions of interest with overlap areas. The arrows indicate image boundaries, which are noticeable due to temporal scattering variation of the nerve fiber features, rather than poor image registration. See full resolution montages and sum maps in Visualization 10 ^{(12.7MB, tiff)} and Visualization 11 ^{(12.7MB, tiff)} , respectively.

8. Summary

A method for tiling retinal images from AO ophthalmoscopes was demonstrated. The workflow performs pairwise registration to establish local alignments, followed by global registration to enforce mosaic-wide consistency; relative translations, rotations, and, if needed, scaling are estimated using the feature-agnostic ZNWCC. The ZNWCC allows the use of binary and real-valued weight masks to account for image irregular boundaries and mitigate the impact of pixels with low SNR and/or affected by motion of features such as blood cells. Each tile or partial montage pair is combined through a contrast-matching weighted average that compensates for differences in mean brightness and contrast across the overlapping regions, optionally using weight masks. The montage is assembled through pairwise registration and contrast matching of tiles and partial montages, providing larger overlapping areas than prior AO retinal image montaging methods that rely on pairwise registration of individual tiles [48,50–54]. Initial montages can be refined by maximizing a global registration metric to mitigate the accumulation of registration errors.

The approach was demonstrated on AOSLO retinal images from humans and mice using a single imaging channel for registration. The absence of duplicate or smeared retinal features in tile overlap regions, together with the constrained set of geometric transformations applied during tiling, demonstrates the value of rigorous distortion correction in preprocessing for preserving retinal anatomy. This approach contrasts with montaging paradigms that apply arbitrary distortions to force alignment across overlapping regions. While such montages may be visually appealing and potentially suitable for psychophysical experiments [94], do not preserve retinal anatomy. Finally, the methodology presented here can operate with multiple imaging channels simultaneously to reduce registration errors and is directly applicable to both coherent and incoherent imaging modalities.

Funding

National Institutes of Health https://ror.org/01cwqze88 ( OT2OD038128); National Eye Institute https://ror.org/03wkg3b53 ( P30EY026877, R01EY031360, R01EY032147, R01EY032669); ARPA-H; Research to Prevent Blindness https://ror.org/04drjs621 ( Departmental Award).

Disclosures

The authors declare no conflicts of interest.

Data Availability

Software and Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.