Decoding memory function through naturalistic gaze patterns

Abstract

Eye movements are closely linked to encoding and retrieval processes, with changes in viewing behavior reflecting age- and pathology-related memory decline. In the current study, we leveraged this relationship to explore possible gaze-based indicators of memory function. Across two task-free viewing experiments, we investigated changes in naturalistic viewing behavior across five participant groups spanning a broad spectrum of memory function, from healthy young adults to amnesic cases. We show that memory decline is associated with an underlying reduction in explorative, adaptive, and differentiated visual sampling of the environment. Our results provide compelling evidence that naturalistic gaze patterns can serve as a sensitive marker of cognitive decline.

Our retina limits visual details primarily to the fovea, requiring us to move our eyes continuously to process our visual environment. Eye movements thus provide a high spatial and temporal resolution, noninvasive measure of how and what we encode into and subsequently retrieve from memory (for review, see refs.1, 2, and 3. Notably, individuals with memory decline resulting from age, disease, or neurological insult show alterations in visual sampling (e.g., fixation count refs. 4–6). Other work indicates that groups differing in memory status diverge across multiple gaze features (e.g., ref.7 and 8) suggesting that univariate gaze metrics may not fully capture the complexity of memory-related viewing behavior. Here, we investigate whether and how multivariate gaze patterns reflect systematic variation in memory function, and what they may reveal about the underlying dynamics of encoding-related visual exploration.

In the present study, we explored changes in naturalistic viewing behavior (using measures previously linked to memory performance and/or hippocampal function, e.g., ref. 9 and 10 across five groups—younger adults (YA), healthy older adults (HOA), individuals at risk for significant cognitive decline (AR), individuals with mild cognitive impairment (MCI), and individuals with amnesia—who are known to differ in memory and/or hippocampal (HC)/ medial temporal lobe (MTL) function (e.g., refs. 11, 12, for review, see ref. 13). In two task-free experiments, participants viewed images either once (Exp 1) or three times (Exp 2) while their eye movements were tracked. Using multivariate gaze analyses (Fig. 1A), we show that eye movements can reveal population differences in memory function, even in the absence of explicit task demands. Specifically, our results indicate that memory decline is associated with reduced visual exploration, less effective updating of encoded representations over repeated viewings, and diminished differentiation of those representations.

Fig. 1.

(A) Fixations belonging to a single participant viewing a single image are spatially smoothed using a Gaussian kernel weighted by fixation duration, and then vectorized. Corresponding vectors are correlated using Fisher z-transformed Pearson correlations. (B) Idiosyncratic gaze similarity is computed as the average of similarity scores obtained by correlating each image with every other image viewed within the same block. Repetitive gaze similarity is computed as the average of similarity scores obtained by correlating each image with itself (over three presentations). (C–E) Conditional effects of group on (C) idiosyncratic gaze similarity, (D) fixation dispersion, and (E) repetitive gaze similarity, from Bayesian linear mixed effects models. Group is coded for both linear (Left) and monotonic (Right) effects. The shaded area (Left) and whiskers (Right) represent the 95% credible intervals for the predicted values.

Results

Experiment 1.

In Exp 1, participants viewed a series of 120 images (10 images from each of 12 categories, see SI Appendix) for 5 s each across three blocks. Each block contained 40 novel images selected randomly from each of three predetermined image sets (four categories per set). Idiosyncratic gaze similarity was computed as the average correlation of eye movements for every image with eye movements for every other image viewed within the same block by the same participant (eyesim package, ref. 10), Fig. 1B. This measure captures the distinctiveness of encoded representations, with low idiosyncratic gaze similarity reflecting an encoding pattern that is unique to a particular image and high similarity reflecting a generic pattern of encoding.

To elucidate the relationship between group status and idiosyncratic gaze similarity, we used Bayesian multilevel modeling (brms package) to model a linear effect of group based on presumed memory function (YA > HOA > AR > MCI > Amnesia). Results of the model revealed a meaningful linear effect (β = 0.007, 95% CI [0.001, 0.010]). The posterior probability of a positive effect was 99%, providing strong evidence that idiosyncratic gaze similarity increased linearly across adjacent groups, with the lowest similarity in the YA group and the highest similarity in the Amnesia group (Fig. 1 C Left).

We subsequently modeled a monotonic effect of group, treating group as an ordinal variable without assuming equal differences between successive groups in terms of memory function. Group status was positively predictive of idiosyncratic gaze similarity (β = 0.006, 95% CI [0.001, 0.012]), posterior probability that β > 0 = 99%, Fig. 1 C Right), and this effect persisted after controlling for broader differences in viewing behavior (SI Appendix). Model comparison revealed an expected log predictive density (ELPD) difference of −0.5 (SE = 0.1), favoring the monotonic model. These results indicate that visual exploration is affected by changes in memory and/or HC/MTL function such that declines are accompanied by less differentiated encoding (gaze) patterns across distinct images.

To further probe changes in visual exploration, we subsequently modeled group differences in the number and dispersion of fixations. The number of fixations increased numerically across groups, but the effect was not reliable in either model (SI Appendix). Fixation dispersion decreased across groups, with posterior evidence supporting both a linear (β = −9.06, 95% CI [−13.629, −4.562], posterior probability that β < 0 = 100%, Fig. 1 D Left) and monotonic effect (β = −8.13, 95% CI [−13.702, −3.864], posterior probability that β < 0 = 99%, Fig. 1 D Right), ELPD difference (SE) = −0.11 (0.17), suggesting that decreasing memory function is associated with a reduction in explorative viewing (see also ref. 8).

Experiment 2.

In Exp 2, participants viewed images (5 s each) across three blocks. Each block contained 120 images (new images taken from the same source as Exp 1), 60 of which were presented once, and 60 of which repeated across blocks (i.e., presented three times total). Analysis of eye movements was restricted to repeated images. Three correlations were computed for each image for each participant (R_Rep1-Rep2, R_Rep1-Rep3, R_Rep2-Rep3) (7) which were averaged, yielding a single score representing the average gaze overlap for each image. Thus, while Exp 1 compared eye movements across unique images (idiosyncratic gaze similarity), Exp 2 compared eye movements across the same image presented repeatedly (repetitive gaze similarity). Repetitive gaze similarity reflects the completeness or richness of an encoded representation, with a low score indicating adaptive encoding (viewing) of different image features with each presentation (i.e., memory updating) and a high score indicating attending to and encoding of the same image features.

Both the linear (β = 0.02, 95% CI [0.002, 0.027], posterior probability that β > 0 = 99%, Fig. 1 E Left) and monotonic models (β = 0.01, 95% CI [0.001, 0.026], posterior probability that β > 0 = 99%, Fig. 1 E Right) revealed a strong increase in repetitive gaze similarity across groups with presumed decreases in memory function, even after controlling for general viewing changes across blocks (SI Appendix), with model comparison slightly favoring the linear model (ELPD difference (SE) = −0.69(0.65). In other words, while healthy YAs encoded unique image features with each presentation, thus building up a more comprehensive representation of each image in memory, participants with decreased memory and/or HC/MTL function tended to view the same image features with each presentation.

Discussion

Research has documented changes in memory-related viewing behavior in healthy individuals and those with cognitive and/or neural compromise (e.g., refs. 4–8), but no study has examined how these changes unfold along a continuum of brain health and memory function. We used multivariate gaze similarity analyses to investigate changes in naturalistic viewing behavior across two experiments in individuals belonging to five groups spanning a broad spectrum of memory and (probable or confirmed) HC/MTL decline. Although we lacked confirmatory neural data for all participants, extensive prior evidence supports the ordering of the current groups based on both memory and HC/MTL function (e.g., refs. 11, 12 for review, see ref. 13). While other cognitive/ brain changes can result in changes to naturalistic viewing, a reanalysis of the data excluding nonamnestic MCI cases (who may have broader damage, see SI Appendix), coupled with the cognitive and neural profiles of the amnesic cases, further suggests that the most parsimonious explanation for the described pattern of results is a change in memory and/or HC/MTL function across groups. Our results show that, even without explicit task demands, gaze patterns varied systematically by group. Specifically, lower memory function was associated with more similar gaze patterns across distinct and repeated images and lower dispersion of eye movements. These findings suggest that suboptimal encoding patterns may underlie the formation of impoverished memory representations in individuals with memory impairment and lay a foundation for future work using multivariate gaze metrics to diagnose and track memory and/or HC/MTL function.

Materials and Methods

Participants included young adults (Exp 1: n = 35, Exp 2: n = 26), healthy older adults (Exp 1: n = 36, Exp 2: n = 28), individuals at risk for significant cognitive decline based on a below threshold score (< 26) on the Montreal Cognitive Assessment (Exp 1: n = 12, Exp 2: n = 10), individuals diagnosed with MCI (Exp 1: n = 15, Exp 2: n = 11), and individuals diagnosed with amnesia (Exp 1: n = 4, Exp 2: n = 3). Eye movements were tracked using an Eyelink II head-mounted eyetracker (SR Research Ltd). All models included random intercepts for participant and image (iterations = 8,000). Model performance was evaluated using leave-one-out cross-validation (LOO) with moment-matching. Where LOO estimates were unreliable, the models were refit. Models were compared using the ELPD based on LOO. Informed consent was obtained from all participants in accordance with the Baycrest Research Ethics Board. For complete study details, see SI Appendix.

Supplementary Material

Appendix 01 (PDF)

Data, Materials, and Software Availability

Anonymized Fixation reports (csv), subject demographics, and model tables have been deposited in OSF (10.17605/OSF.IO/GKNVQ) (14).