A novel computational model to probe visual search deficits during motor performance

Visual search has traditionally been studied in cognitive and perceptual paradigms, but little is known about how it contributes to visuomotor performance. We have developed a novel computational model to examine how three underlying processes of visual search (spatial planning, working memory, and peripheral visual processing) contribute to visual search during a visuomotor task. We show that deficits integrating spatial planning and working memory underlie abnormal performance in stroke survivors with frontoparietal damage.

Abstract

Successful execution of many motor skills relies on well-organized visual search (voluntary eye movements that actively scan the environment for task-relevant information). Although impairments of visual search that result from brain injuries are linked to diminished motor performance, the neural processes that guide visual search within this context remain largely unknown. The first objective of this study was to examine how visual search in healthy adults and stroke survivors is used to guide hand movements during the Trail Making Test (TMT), a neuropsychological task that is a strong predictor of visuomotor and cognitive deficits. Our second objective was to develop a novel computational model to investigate combinatorial interactions between three underlying processes of visual search (spatial planning, working memory, and peripheral visual processing). We predicted that stroke survivors would exhibit deficits in integrating the three underlying processes, resulting in deteriorated overall task performance. We found that normal TMT performance is associated with patterns of visual search that primarily rely on spatial planning and/or working memory (but not peripheral visual processing). Our computational model suggested that abnormal TMT performance following stroke is associated with impairments of visual search that are characterized by deficits integrating spatial planning and working memory. This innovative methodology provides a novel framework for studying how the neural processes underlying visual search interact combinatorially to guide motor performance.

NEW & NOTEWORTHY Visual search has traditionally been studied in cognitive and perceptual paradigms, but little is known about how it contributes to visuomotor performance. We have developed a novel computational model to examine how three underlying processes of visual search (spatial planning, working memory, and peripheral visual processing) contribute to visual search during a visuomotor task. We show that deficits integrating spatial planning and working memory underlie abnormal performance in stroke survivors with frontoparietal damage.

many daily activities, such as driving (Mourant and Rockwell 1972), food preparation (Hayhoe et al. 2003; Land et al. 1999), and sports (Land and McLeod 2000; Williams et al. 1994), require organized patterns of eye movements (visual search) to actively scan the environment for information that guides limb and body movements (Hayhoe and Ballard 2005). Visual search is a multifaceted behavior that uses top-down cognitive processes to organize and integrate perceptual processes involved in vision with oculomotor processes involved in controlling eye movements (Chen and Zelinsky 2006; Land and Horwood 1995; Nash et al. 2016; Shinoda et al. 2001). The distinct cognitive and perceptual processes involved in visual search have been well studied in the reading and scene-perception literature, but we still have a limited understanding of how these processes interact to produce patterns of visual search that guide limb movements.

Investigating the interactions between visual search and limb movements is especially important from a clinical viewpoint because lesions and/or diminished connectivity involving a broad network of brain regions can produce functional deficits (Corbetta et al. 2005; Middleton and Strick 2000; Siegel et al. 2016). The importance of visual search in functional performance is further highlighted by a recent study showing that stroke survivors with mild motor impairments and no visuospatial neglect perform similar to age-matched controls when driving in sparse traffic but make significantly more driving errors in heavy traffic (Hird et al. 2015). This suggests that visuomotor performance is disrupted in complex visual environments that necessitate organized visual search. However, very little is known about how visual search and its underlying cognitive and perceptual processes are affected in clinical populations. To better understand the relationship between visual search and visuomotor performance following stroke, we need innovative methodologies for studying the underlying mechanisms that guide visual search during visuomotor performance. Traditional models of visual search (Treisman and Gelade 1980; Wolfe 1994a) provide a basis for developing novel task-specific computational models to better understand how normal and impaired visual search guide visuomotor performance.

The first objective of this study was to develop an innovative methodology for investigating the extent to which stroke-induced deficits in cognitive and visual processing contribute to impairments of visual search during visuomotor performance. To address this objective, we used an upper-limb robot with integrated eye tracking (Fig. 1A) to examine healthy adults and stroke survivors who performed the Trail Making Test (TMT; Reitan 1958), a classic neuropsychological test used to assess deficits in visuomotor processing and executive function (Arbuthnott and Frank 2000; Kopp et al. 2015). TMT consists of two parts, a simpler numeric variant, TMT-A (Fig. 1B), and a cognitively challenging alphanumeric variant, TMT-B (Fig. 1C). Like the children's game of connect the dots, both variants require participants to sequentially locate and make reaching movements to draw lines between 25 sequential targets (1, 2, 3, . . ., 25 in TMT-A and alternating numbers and letters 1, A, 2, B, 3, C, . . ., 13 in TMT-B) that are pseudorandomly distributed within the workspace. Subjects are instructed to perform the test in the shortest possible time. Before making a reaching movement to a target, participants perform a series of saccades and fixations (visual search) to locate the next target(s) in the sequence. TMT performance is known to be correlated with executive planning (Wölwer and Gaebel 2002), visual search (Crowe 1998; Salthouse et al. 2000), working memory (Salthouse 2011; Sanchez-Cubillo et al. 2009), and oculomotor speed (Wölwer and Gaebel 2003). These attributes of the TMT also make it a strong predictor of driving performance following stroke (Devos et al. 2011).

Fig. 1.

Robotic apparatus (A) and TMT task (B and C). A: participant grasps the robotic handle while sitting in the custom chair. Targets are displayed in the same horizontal plane as the hands. The remote eye tracker is located at the back of the workspace. B and C: top view of the virtual environment showing the target workspace for TMT-A (B) and TMT-B (C). Hand location is displayed as a white circle, and black lines show previously traversed targets; both are visible to the participant. The arm is drawn for illustrative purposes only and was not visible to the participant.

We leveraged TMT to examine two cognitive processes (spatial planning and working memory) and one perceptual process (peripheral visual processing) that we expected would guide visual search during visuomotor performance. Spatial planning is a cognitive process that is crucial for organizing the spatial distribution of saccades when searching through visual scenes that are spatially organized in a rule-governed manner (Neider and Zelinsky 2006; Shinoda et al. 2001; Wolfe et al. 2011; Woods et al. 2013). In TMT, targets are spatially organized such that targets near each other in the numeric/alphanumeric sequence are likely to be located in spatial proximity. Accordingly, spatial planning of visual search during TMT should prioritize nearest neighbors rather than randomly searching the workspace (Fig. 2A). Working memory is another cognitive process that may contribute to efficient visual search. In TMT, subjects may hold a small number of recently fixated targets in working memory and use this information to optimize their search (Fig. 2B). Specifically, the use of working memory during TMT can be used to maximize saccades back to a recently fixated target when it is the next target in the sequence and to minimize saccades back to recently fixated targets that are not the next target in the sequence (Peterson et al. 2001). However, some studies have found that using working memory can reduce visual search efficiency (Kane et al. 2006; Woodman and Luck 2004), and thus use of working memory may be task dependent. Peripheral visual processing is used to gather information and guide visual search without overtly fixating targets (Ludwig et al. 2014; Weiss et al. 2014). In TMT, while searching for the next target in the sequence, subjects may use peripheral vision to avoid saccades back to targets that were previously connected by reaching movements (reached targets were connected by a salient black line; Fig. 2C). Studies have shown that peripheral visual processing is frequently used by experienced performers in many activities of daily living (Ando et al. 2001; Summala et al. 1996). However, during unconstrained visual search, peripheral visual processing may be suppressed in favor of directly fixating objects of interest (Findlay and Gilchrist 1998).

Fig. 2.

The 3 proposed underlying processes of visual search. A: spatial planning. Two hypothetical topographic spans (dotted orange circles) illustrate how variations of spatial planning can guide visual search. The participant is searching for “16” while the hand is at “15” (shown in red). The smaller search span (gray numbers) is more efficient than the larger span (blue and gray numbers combined). B: working memory. Example shows how a hypothetical working memory buffer of size 4 would guide visual search. The 4 previously fixated targets (purple targets: 8, 10, 14, and 18) are stored in working memory and would be excluded from the topographic span for the next saccade to search for “16.” C: peripheral visual processing. Hand paths through previously traversed targets (black lines) stayed on the visual display, allowing participants to use peripheral vision to exclude previously traversed targets (red numbers) from the topographic span used to search for “16.”

In the present study, we investigated the extent to which healthy controls and stroke survivors use spatial planning, working memory, and peripheral visual processing while searching for targets during TMT performance. First, during each search for the next target, we predicted that healthy controls would use spatial planning to constrain their visual search to nearby spatial neighbors of the current target. Second, we predicted that healthy controls would use working memory to inhibit visual search from returning back to recently fixated targets that are not the next target in the sequence. Third, we predicted that healthy controls would use peripheral visual processing of lines between targets to inhibit their visual search from returning to targets that were previously connected by reaching movements. Fourth, in contrast to healthy controls, we predicted that stroke survivors would exhibit deficits integrating spatial planning, working memory, and peripheral visual processing, resulting in impaired visual search and diminished TMT performance. Fifth, we predicted that when the cognitive demands of the task are high during TMT-B, both healthy controls and stroke survivors would decrease their use of these three processes and perform visual search in a more unstructured fashion (Alvarez and Cavanagh 2004; Eng et al. 2005). Finally, we predicted that this effect would be greater in stroke survivors; i.e., the relative performance of stroke survivors would be much worse in TMT-B than in TMT-A.

The second objective of this study was to develop a novel computational model of visual search for examining how deficits in spatial planning, working memory, and peripheral visual processing interact with each other to produce task-specific impairments of visual search in clinical populations. These three processes provide a comprehensive framework for investigating visual search, thereby providing an innovative methodology for investigating and interpreting how interactions between these processes are affected in stroke survivors. In this article, we focus on the visual search behavior and not the reaching movements.

MATERIALS AND METHODS

Experimental setup.

Experiments were performed on an upper-limb robot (KINARM EndPoint Lab; BKIN Technologies, Kingston, ON, Canada) with integrated eye tracking (Eyelink 1000; SR Research, Ottawa, ON, Canada) and an augmented reality display (Fig. 1A). The SR Eyelink 1000 Remote is a monocular system that does not use a chinrest and is calibrated for the two-dimensional (2-D) horizontal workspace using proprietary algorithms (BKIN Technologies). The maximum sampling frequency of the Remote system is 500 Hz, and accuracy is 0.5°. Hand kinematics from the robot (hand position, velocity, and acceleration) were sampled at 1,000 Hz, and gaze kinematics from the eye tracker were sampled at 500 Hz. Gaze data were then interpolated and upsampled at 1,000 Hz offline using MATLAB (version 8.1; The MathWorks, Natick, MA).

Participants made reaching movements in the horizontal plane while sitting in a customized chair with adjustable height. Visual targets were displayed in the same horizontal plane as the hands, allowing the subjects to interact realistically with the visual targets. The eye tracker could accurately monitor eye movements within a workspace that covered an area of ∼50 cm wide × 50 cm deep located in the middle of the KINARM workspace. This workspace corresponded to a visual area of ∼55° wide × 40° deep. However, the mapping between the Cartesian coordinates of the KINARM workspace and the corresponding angular coordinates of visual space was nonlinear because visual stimuli were presented in the horizontal plane. As a result, numbers/letters closer to subjects encompassed a larger visual angle in degrees than numbers/letters further away (Fig. 1, B and C). The arms and hands were occluded from vision by a shield and fabric cover. Visual feedback of hand position was provided by a small white circle (1-cm diameter) projected on the display.

Experimental task.

As described in the Introduction, TMT is a classic neuropsychological test that requires organized visual search, quick decision-making, and rapid, accurate reaching movements. In TMT-A, participants move their hand to draw lines connecting the first 25 positive natural numbers (1, 2, 3, . . ., 25). In TMT-B, which is more cognitively challenging, participants draw lines alternating between the first 13 positive natural numbers and the first 12 Roman letters (1, A, 2, B, 3, C, . . ., 13; Fig. 1, B and C). Participants were instructed to complete the tests as quickly as possible. Control participants used their dominant hand, and stroke survivors used their less impaired hand, which helped minimize confounds due to motor impairments.

Before completing the actual TMT-A and TMT-B tests, participants completed sequences of five practice targets for TMT-A (1, 2, 3, 4, 5) and TMT-B (1, A, 2, B, 3). As typically done in the clinic, TMT-A was always performed before TMT-B (Crowe 1998). Many studies have shown that the TMT is susceptible to practice effects that make it difficult to parse out cognitive and perceptual differences between clinical populations and healthy controls (Bowie and Harvey 2006; Buck et al. 2008; Dye 1979). Therefore, similar to the clinical version of the test, only one trial each of TMT-A and TMT-B was performed by the participants. The trajectory of the completed portion of the test also remained on the display for the entire duration of the trial (black lines in Fig. 1, B and C). Furthermore, if participants made an error by moving their hand to an incorrect number/letter, the previous number/letter turned red and the participants returned their hand to the previous number/letter before continuing with the test. The robot neither assisted nor resisted any movements during the task.

Participants.

Sixty-nine participants volunteered for the study, including 37 young adults, 16 older adults, and 16 stroke survivors (Table 1). Two control groups (young and older) were included to quantify normal visual search and also age-induced deficits in visual search. This allowed us to minimize confounds due to age-induced cognitive impairments that could affect visual search. All stroke survivors had a stroke affecting their left cortical middle cerebral artery (MCA) territory. We collected neuroimaging data from 12 stroke survivors and measured the lesion volumes in two areas of the frontoparietal network (middle frontal gyrus and superior parietal gyrus). All scans were acquired using a Siemens Trio 3-Tesla magnetic resonance imaging scanner with a 12-element head coil. The scans confirmed that the stroke survivors had substantial damage in the frontoparietal network.

Table 1.

Demographic information for participants

	Young	Older	Stroke Survivors
No. of participants	37	16	16
Age, yr	33 [21–50]	60 [52–70]	58 [38–74]
Box and Block score	72 [48–94]	68 [50–82]	58 [43–76]
ViCA score	19 [16–20]	18 [16–20]	15 [12–20]

Box and Block scores are reported for the dominant hand (controls) and less affected hand (stroke survivors); numbers are reported as mean [range]. Maximum ViCA score is 20.

Stroke survivors were included if they had 1) a single unilateral stroke at least 6 mo before testing and 2) difficulty performing one or more relevant activities of daily living (Stroke Impact Scale-16, one or more individual scores <5). Control participants and stroke survivors were excluded if they had 1) history of a central or peripheral neurological disorder (other than stroke), 2) a musculoskeletal problem of the tested upper extremity, 3) moderate to severe spasticity of the tested upper extremity (Modified Ashworth Scale score ≥2), 4) an uncorrected visual impairment (Snellen chart and confrontation testing), 5) evidence of visuospatial neglect (line bisection and letter cancellation), or 6) difficulty understanding and following simple instructions. The Institutional Review Board of the University of South Carolina approved the study. All participants provided informed consent before participating in the study.

Cognitive impairments were assessed using an in-house cognitive screening tool, Visual Cognition Assessment (ViCA). This is a visuospatial analog of the Montreal Cognitive Assessment (MoCA; Nasreddine et al. 2005) that uses nonverbal responses for all non-language components of the MoCA (visuospatial/executive, memory, attention, and orientation).

Gaze data processing and event identification.

Details of gaze processing and gaze-event identification have been recently published (Singh et al. 2016). In brief, data from the robot and gaze-tracking systems were low-pass filtered at 20 Hz. Gaze data were preprocessed to remove blinks (due to closed eyelids), one-sample spikes (due to incorrect detection of corneal reflection), and screen outliers (due to instances when gaze drifts outside the workspace). Our main interest was in establishing the times of initiation and termination of saccades and fixations, and the 20-Hz cutoff frequency adequately removed high-frequency oscillations during fixations to accurately mark the points when saccades were initiated and terminated. Gaze events were identified as saccades and fixations using adaptive velocity and acceleration thresholds (see Fig. 10 in Singh et al. 2016). In particular, our previous analyses showed that velocity thresholds vary substantially between participants (25–90°/s) but that acceleration threshold is relatively constant (6,000°/s²). For each velocity peak that exceeded the velocity threshold, we confirmed that the peak acceleration leading up to the velocity peak also exceeded the acceleration threshold (6,000°/s²). If both thresholds were exceeded, we classified the gaze event as a saccade. For each saccade, we found the first inflection point before and after the local peak in gaze angular velocity. Saccade onset corresponded to the first inflection point before the local peak in gaze angular velocity. Saccade offset (fixation onset) was determined by starting at the first inflection point after the local peak in gaze angular velocity and finding the first point in time at which the gaze velocity and acceleration remained continuously lower than the respective thresholds for at least 40 ms. The MATLAB code for computing the adaptive thresholds is available online as supplementary material for our previous publication (Singh et al. 2016). For each fixation, a region of interest was computed as a 2-D perimeter (convex hull) around all the gaze points using the convhull function in MATLAB. We then computed the centroid of the 2-D perimeter, and only those numbers/letters that fell within a 2° radius of the centroid were assumed to be overtly foveated during fixations. Conversely, we assumed that numbers/letters located outside of the 2° radius of the centroid fell within peripheral vision. Typically, participants fixated on only one number/letter (∼95% cases). When participants simultaneously fixated more than one number/letter, we assumed that they recognized both of them.

Experimental measures.

Measures were divided into three categories: measures of task performance, visual search, and underlying processes. As with the pencil-paper version of TMT that is used clinically, measures of task performance included total time (total time to complete the test) and reaching errors (reaching movements made to a number/letter in incorrect sequence). Measures of visual search included total fixations (total number of fixations on number/letters in a trial), average saccade duration, and average fixation duration. We computed four measures designed to reflect the underlying processes of visual search. These measures included 1) search distance, the sum of Euclidean distances of fixations from the current hand-dwell location; 2) unique fixations, the average number of unique numbers/letters that were fixated between two hand dwells on sequential targets (repeated fixations to the same target were not counted); 3) saccades between refixations, the number of saccades to different numbers/letters between repeat fixations of a target; and 4) refixations on traversed targets, repeat fixations on targets that had already been traversed by the hand (i.e., targets connected by black lines). Together, a smaller search distance and fewer unique fixations reflect efficient spatial planning. A high number of saccades between refixations indicates effective use of working memory. Finally, few refixations on traversed targets reflects the use of peripheral visual processing to avoid repeat fixations on targets connected by lines after they were traversed by the hand.

Computational model of visual search.

We created a stochastic model of visual search with three free parameters to quantify spatial planning, working memory, and peripheral visual processing. The model allowed us to quantify how the three cognitive/perceptual processes interact with each other during visual search. For spatial planning, the approximate radius of the visual search (Fig. 2A) was modeled as the topographic span, TS, whose elements consisted of the nearest (in an ordinal sense) numeric (TMT-A) or alphanumeric (TMT-B) sequence of numbers/letters to the current target. This is a simplification that does not represent the exact “spatial” nature of TMT, but this was a good first approximation given that the nearest sequence of numbers/letters were typically nearest targets in terms of spatial proximity. TS could vary from 1 element (search only the nearest neighbor closest to the hand) to 24 elements (search randomly within the entire workspace). To illustrate how the model worked, let us assume that a participant was looking for “16” in TMT-A while his/her hand was resting on “15.” If TS was 7, the elements within TS would consist of numbers 12–14, 16–19. The model would randomly search for 16 within this vector until it was found. Once a target was found, the model assumed that the hand promptly moved to the target. The model-predicted time for the hand to reach a target i in the 25-target sequence was

$$\begin{array}{l}{\left({\text{T}}_{\text{i}}^{\text{Model}}\right)}_{\text{TS,WM,PP}}={\left({\text{T}}_{\text{i}-\text{1}}^{\text{Model}}\right)}_{\text{TS,WM,PP}}+{\{\underset{\text{j}=1}{\overset{{\text{n}}_{\text{i}}}{\mathrm{\Sigma }}}\hspace{0.17em}{\displaystyle }{\text{T}}_{j,\text{saccade}}+{\text{T}}_{j,\text{fixation}}\}}_{\text{TS,WM,PP}}+\left({\text{T}}_{i\text{,movement\_time}}\right),\end{array}$$ (1)

where i varied from 2 to 25 and T₁^Model = 0. The first term on the right-hand side in Eq. 1 indicates the time at which the simulation predicted that the hand reached the i − 1-th target. The second term on the right-hand side is the visual search time. The subscripts (TS, WM, PP) indicate that the search was performed within the search span for a fixed value of TS, WM, and PP (see details for WM and PP below). n_i is the number of saccades (same as number of fixations) till the i-th target was found, and because of the stochastic nature of the search, n_i was different for different targets. Each saccade and fixation duration was sampled from two different Weibull distributions (Over et al. 2006). The probability density function of a Weibull random variable is given as

$$\begin{array}{cc}\text{p}(\text{x}\hspace{0.17em}|\hspace{0.17em}\alpha ,\beta )=\frac{\beta }{\alpha }{\left(\frac{\text{x}}{\alpha }\right)}^{\beta -1}{\text{e}}^{-{\left(\frac{\text{x}}{\alpha }\right)}^{\beta }}& x\ge 0\\ \hfill 0& \hfill x<0\end{array}.$$ (2)

The Weibull parameters α and β were obtained for saccades and fixation durations separately by fitting Weibull distributions to the experimental data of an individual participant. Once the target was found in the simulation, a hand movement time (T_{i,movement_time}) sampled from a uniform distribution was added to the visual search time (third term in the right-hand side of Eq. 1). The upper and lower limits of the distribution were the shortest and longest hand movement times in the experimental trial. This process was repeated for all i.

Working memory was modeled as a first in-first out memory buffer. Studies have shown that healthy adults can store 7 ± 2 items and 6 ± 2 spatial locations in working memory (Cowan 2010; Lisman and Idiart 1995; Miller 1956). Given that targets were numbers/letters in specific spatial locations, our index of working memory, WM, could vary from 0 (working memory not used) to 8 (memory buffer of 8 simultaneous numbers/letters). The upper limit of 8 was found to be sufficient for our sampled population. The buffer represents the most recently fixated numbers/letters. The elements within this buffer were excluded from a larger set of elements within TS during each search for the next target in the sequence. For example, with a WM of 4, the last four fixated numbers/letters were ignored from the search span. TS from the example stated above would then be effectively reduced from seven to three elements. The search is again conducted randomly within these three numbers. For example, during a search for “16,” if [12, 13, 17, 19] is the WM buffer of the four most recently fixated numbers (in that order), then during the next iteration, the model would search for 16 within the vector [14, 16, 18]. If “14” was the next fixated item, then the WM buffer would become [13, 17, 19, 14] and the model would randomly search for 16 within the vector [12, 16, 18]. This process would continue till 16 was found. Also, the WM buffer was continuous across time, thus it was not reset (emptied) after a target was found. This implicitly assumed that the memory buffer was not affected by reaching movements.

Peripheral visual processing was modeled as a binary variable. An index, PP, was set to 0 if this process was not used and 1 otherwise. If PP equaled 1 (and WM was empty), then for the example stated above, the model performed a random search within a sequence of 7 numbers starting at “16” (i.e., search span consisted of numbers 16–22). If PP equaled 0, the search span would have been centered on the current hand location of “15” (i.e., numbers 12–14, 16–19). PP was combined with TS and WM such that TS was always reduced by the extent of overlap with the memory buffer. Both working memory and peripheral visual processing contribute to avoiding wasteful saccades to recently fixated (WM > 0) or traversed (PP = 1) targets. However, the model allows for the possibility that only one process is used by a participant (e.g., WM = 0, PP = 1 or WM > 0, PP = 0).

For each combination of TS, WM, and PP, the model accounted for decreasing task difficulty as the trial progressed. For example, TS = 7, WM = 0, and PP = 1 imply that the model searches for the next target in a search span of 7 numbers/letters while not remembering any previously fixated number/letter and ignoring all targets that the hand has already traversed. Therefore, if the current hand location in the model is “22” in TMT-A, then the model searches for “23” within a search span of only 3 elements (23, 24, and 25). If, on the other hand, PP = 0 (same TS and WM), then the search span would contain 7 elements (18, 19, 20, 21, 23, 24, and 25).

Once T_TS,WM,PP^Model (25 × 1 vector) was obtained for a given combination of TS, WM, and PP (Eq. 1), we repeated the process 100 times and computed the average, T̄_TS,WM,PP^Model, of the 100 simulated searches of the 25-target sequence. Thus each element of T̄_TS,WM,PP^Model is an average over 100 repetitions of the simulation. This process was repeated for all different combinations of TS [1 to 24], WM [0 to 8], and PP [0, 1]. The combination of TS, WM, and PP values that minimized the cost function in Eq. 3 was extracted separately for each TMT-A and TMT-B trial. In terms of model implementation, there were no differences between TMT-A and TMT-B except for the inputs (fixation, saccade, and hand movement durations) to the model that were used to compute the parameters for the Weibull and uniform distributions:

$$J=\underset{i=1}{\overset{25}{\mathrm{\Sigma }}}\hspace{0.17em}{\displaystyle }{\left({\left\{{\overline{\text{T}}}_{\text{i}}^{\text{Model}}\right\}}_{\text{TS,WM,PP}}-{\text{T}}_{\text{i}}^{\text{Experiment}}\right)}^2.$$ (3)

To validate the model, we correlated the three model parameters with the experimental measures of spatial planning (search distance, unique fixations), working memory (saccades between refixations), and peripheral visual processing (refixations on traversed targets). For spatial planning, we performed linear regression. For working memory and peripheral visual processing, we performed logistic regression because the model-generated values for WM and PP were within a very narrow range of values (0–8 for WM and 0–1 for PP). Finally, using the best set of model parameters obtained from Eq. 3, we quantitatively compared the model-predicted time course of completion of a trial and the experimental trial. We reiterate that none of the model parameters (TS, WM, and PP) were explicitly fit to the experimental measures of spatial planning (search distance, unique fixations), working memory (saccades between refixations) and peripheral visual processing (refixations on traversed targets). Rather, they emerged from fitting the model to the overall trial durations obtained from each individual trial for a subject (Eq. 3).

Statistics.

Descriptive statistics are presented in the text and figures as means ± SE. We performed a 2 × 3 linear mixed-effects analysis of the relationship between Task (TMT-A, TMT-B) and Group (young, older, stroke). We modeled Task and Group (with interaction term) as fixed effects. Participants and their intercepts were modeled as random effects nested within Group. Whenever visual inspection of residual plots revealed clear deviations from either homoscedasticity or normality, variables were log transformed. P values were obtained by likelihood ratio tests of the full model with the effect in question against the model without the effect in question. Tukey's correction was used for multiple comparisons. The level of significance was chosen as α = 0.05. Statistical tests were performed using lme4 in the software package R (Bates et al. 2012).

RESULTS

Figure 3, A–C, shows the spatial distribution of saccades and fixations for a representative participant from each group. The number of fixations made by participants increased progressively from young controls (Fig. 3A) to older controls (Fig. 3B) to stroke survivors (Fig. 3C). Participants also made more fixations in TMT-B than in TMT-A (TMT-B data not shown in Fig. 3). Figure 3, D–F, shows the temporal profiles of the trials shown in Fig. 3, A–C, with targets arranged linearly along the x-axis. These plots highlight differences between stroke survivors and controls in their use of spatial planning, working memory and peripheral visual processing. Differences in spatial planning are evident in the range of numbers that the representative subjects fixated between reaches to successive targets. The healthy controls typically restricted their visual search to a narrow range of numbers (smaller spatial area). In contrast, the representative stroke survivor typically made fixations over a broad range of numbers (greater spatial area). Differences in working memory are reflected in the number of fixations of other targets that intervened between repeat fixations of the same target. The healthy controls typically fixated several other targets between repeat fixations of the same target. In contrast, the stroke survivor often made several repeat fixations of the same target in short succession with few intervening fixations of other targets. This indicates that the healthy controls used working memory more effectively than stroke survivors. Differences in peripheral visual processing are evident in the frequency that the representative subjects fixated previously traversed targets (floating circles). The healthy controls typically made few fixations on previously traversed targets, whereas the stroke survivor made several fixations on previously traversed targets. Although not shown in Fig. 3, participants in all three groups exhibited worse use of spatial planning (searched within larger search spans), working memory (remembered fewer recently fixated numbers/letters), and peripheral visual processing (higher number of fixations to traversed targets) in TMT-B than in TMT-A.

Fig. 3.

Gaze-tracking data for 3 representative participants in TMT-A. A–C: scan paths showing saccades (gold) and fixations (red) for a representative participant. D–F: stem plots of the gaze events and reaching movements for the trials shown in A–C. Targets are plotted sequentially (i.e., 1, 2, 3, . . .) on the X-axis, and total time is plotted on the Y-axis. Green circles indicate fixations, and their sizes indicate relative fixation durations. Blue squares indicate when the hand reached a correct target. Red diamonds indicate when the hand reached an incorrect target (errors). Notice the difference in times to complete the test by the 3 participants. Spatial planning can be visualized by counting the number of circles on all stems between parallel lines passing through any 2 consecutive squares. For example, the red lines in D indicate that only 2 targets (19 and 6) were fixated by the young control while searching for “6.” Working memory can be visualized similarly by drawing parallel lines between any 2 consecutive circles on the same stem and counting all other circles within the lines. For example, the purple lines in E show that several targets were fixated (10, 12, 15, 16, 18, and 21) between repeat fixations on “20” by the older control. Peripheral visual processing can be visualized by counting the floating circles on top of the blue squares (F), which show fixations on previously traversed targets that were connected by lines. Data for TMT-B (not shown) were qualitatively similar.

In addition to using peripheral visual processing to identify lines that connected previously traversed targets, it is also possible that participants could have used peripheral visual processing to identify the targets located in the visual periphery (e.g., 7 looks very different from 3, 8, and 9). Notice in Fig. 3A that the participant makes a reaching movement to “7” without overtly fixating on it first. Overall, only 2.4% of reaching movements were made to targets that were not overtly fixated before reaching. Although this feature of peripheral visual processing is also important to understand, modeling this behavior is difficult and was outside the scope of the current report. Our behavioral analysis and computational modeling of peripheral visual processing focused only on investigating if participants used the lines connecting previously traversed targets as visual landmarks to direct their visual search.

Influence of stroke on overall task performance.

Figure 4A and Table 2 show the group data and the main and interaction effects for total time, respectively. Stroke survivors took longer to complete TMT-A and TMT-B (55.9 ± 7.5 and 118.9 ± 17.2 s) than young (24.8 ± 1.1 and 36.9 ± 1.9 s) and older controls (30.5 ± 1.6 and 51.9 ± 4.3 s) (post hoc comparisons, P < 0.05). In particular, total time for stroke survivors in TMT-B was much longer than for the two control groups (significant main effect of group). The main effect of task was also significant, because the participants in all three groups took longer to complete TMT-B (59.4 ± 5.8 s) than TMT-A (33.4 ± 2.4 s). Controls made fewer reaching errors than stroke survivors, who made many reaching errors, particularly in TMT-B (Fig. 4B). Although the main effects of group and task were significant (Table 2), post hoc tests only revealed a significant difference between young controls and stroke survivors in TMT-B.

Fig. 4.

Individual and group data for measures of task performance. A: log of total time for each group in TMT-A and TMT-B (actual means are displayed above each plot). B: number of reaching errors. In each panel, + and × symbols indicate individual participants for TMT-A and TMT-B, respectively, black horizontal lines indicate the means, dark and light gray areas indicate 1 SE and SD. ⋆ and ⭐ indicate significant differences compared with the young and older groups, respectively.

Table 2.

χ² and P values for main effects (group and task) and interactions (group × task)

	Main Effects				Interaction
	Group		Task		Group × Task
Measure	χ2	P	χ2	P	χ2	P
Total time	49.5	<0.001	69.3	<0.001	11.8	<0.01
Reaching errors	23.8	<0.001	28.4	<0.001	8.7	<0.05
Total fixations	44.9	<0.001	57.3	<0.001	11.6	<0.001
Saccade durations	4.2	>0.05	1.9	>0.05	1.9	>0.05
Fixation durations	4.3	>0.05	3.6	>0.05	2.1	>0.05
Search distance	34.8	<0.001	28.8	<0.001	9.5	<0.01
Unique fixations	32.6	<0.001	43.3	<0.001	5.6	0.05
Saccades between refixations	15.6	<0.01	19.0	<0.001	5.6	0.05
Refixations on traversed targets	14.7	<0.01	48.6	<0.001	1.3	>0.05

Influence of stroke on visual search.

Group data and the statistical results for measures of visual search are shown in Fig. 5 and Table 2, respectively. Figure 5A shows that, on average, stroke survivors made more fixations in TMT-A and TMT-B (139.4 ± 16.8 and 286.5 ± 37.9) than young controls (70.7 ± 3.6 and 95.9 ± 4.9) and older controls (79.0 ± 4.8 and 130.3 ± 8.5) (post hoc comparisons, P < 0.05). In particular, stroke survivors made a disproportionately higher number of fixations in TMT-B than the two control groups (significant main effect of group). The main effect of task was also significant, because the participants from all three groups made more fixations in TMT-B (148.1 ± 13.1) than in TMT-A (88.5 ± 5.6). Average saccade (Fig. 5B) and fixation durations (Fig. 5C) were similar across the three groups and two tasks (see Table 2). This suggests that group differences in task performance reflected differences in cognitive and peripheral visual processes that regulate the frequency and location of fixations rather than differences in oculomotor control (saccade durations) and central visual processing (fixation durations).

Fig. 5.

Individual and group data for visual search measures. A: log of total fixations for each group in TMT-A and TMT-B (actual means are displayed above each plot). B: average saccade durations. C: average fixation durations. Symbols, lines, and shaded areas in A–C are as defined in Fig. 4.

Influence of stroke on underlying processes of visual search.

Figure 6 shows the four empirical measures of the underlying processes of visual search. Spatial planning is illustrated with search distance (Fig. 6A) and unique fixations (Fig. 6B). In TMT-A, both measures were much larger for stroke survivors than controls (Table 2, main effect of group; post hoc comparisons, P < 0.01), indicating that stroke survivors employed poor spatial planning that covered a larger area and was less well organized. Working memory is displayed with saccades between refixations (Fig. 6C). In TMT-A, stroke survivors made fewer saccades to other targets between repeat fixations of the same target (Table 2, main effect of group; post hoc comparisons, P < 0.01), indicating that they used less working memory than controls. In TMT-B, controls made fewer saccades between refixations with values similar to those of stroke survivors (post hoc comparisons, P > 0.05). Peripheral visual processing is shown with refixations on traversed targets (Fig. 6D). In TMT-A, stroke survivors made more repeat fixations on targets that were connected by lines after they were traversed with a hand movement (Table 2, main effect of group; post hoc comparisons, P < 0.01), indicating that they used less peripheral visual processing than controls. Grouped together, all participants exhibited a greater search distance, more unique fixations, fewer saccades between refixations, and more fixations of traversed targets in TMT-B than in TMT-A. This indicates that participants performed visual search in a more unstructured fashion when the cognitive load was higher in TMT-B. Furthermore, more than 10% of total fixations were on previously traversed targets in most participants (Fig. 6D). On average, in TMT-A, young controls, older controls, and stroke survivors made 10.4 (24.2 in TMT-B), 11 (36.8 in TMT-B), and 31.3 (100.5 in TMT-B) saccades back to traversed targets, respectively.

Fig. 6.

Individual and group data for the underlying processes of visual search. A: search distance (spatial planning). B: number of unique fixations (spatial planning). C: saccades between refixations (working memory). D: refixations on traversed targets (peripheral visual processing). The numbers shown in D were computed as percentages of the total number of fixations. Symbols, lines, and shaded areas in A–D are as defined in Fig. 4.

How does visual search affect overall performance?

We examined the extent to which measures of visual search were predictive of task performance (total time). Number of fixations was a strong predictor of task performance (Fig. 7A; Table 3), whereas fixation and saccade durations were weak predictors (Fig. 7, B and C; Table 3).

Fig. 7.

Correlations between measures of visual search and task performance (total time). A: number of fixations. B: average fixation duration. C: average saccade duration. Data from the trials for TMT-A and TMT-B are combined.

Table 3.

Correlations between total time and visual search measures

	Controls		Stroke Survivors
Measure	R2	P	R2	P
Number of fixations	0.86	<0.001	0.95	<.001
Saccade duration	0.00	>0.05	0.00	>0.05
Fixation duration	0.00	>0.05	0.06	>0.05
Search distance	0.49	<0.001	0.18	<0.05
Unique fixations	0.63	<0.001	0.24	<0.01
Saccades between refixations	0.03	>0.05	0.05	>0.05
Refixations on traversed targets	0.37	<0.001	0.01	>0.05

By computing the average number of saccades made per unit time, we explored the possibility that the longer times to complete the tests could have contributed to the higher number of saccades (or fixations) made by the stroke survivors. The group averages for saccades per unit time were 2.71 ± 0.07 (young), 2.57 ± 0.09 (older), and 2.54 ± 0.08 (stroke). Although it seems that older adults and stroke survivors made fewer saccades per unit time, statistical tests showed no differences between the groups. This result strongly suggests a causal relationship between number of fixations and total time, i.e., higher number of fixations contributed to longer total times to complete the test.

We also examined how the underlying processes of visual search predicted overall task performance. The measures of spatial planning (search distance and unique fixations) were good predictors of total time (Table 3). In contrast, saccades between refixations, which was our measure of working memory, was a poor predictor of total time (Table 3). Finally, the number of refixations on traversed targets, which reflects peripheral visual processing, was a strong predictor for the controls but not for the stroke survivors (Table 3).

Computational model of the underlying processes of visual search.

Figure 8A shows the model predictions for a representative participant for different combinations of TS, WM, and PP. If the participant did not use any of the underlying processes of visual search (TS = 24, WM = 0, PP = 0), the model predicted that the participant would have completed the test with a total time of 159 s (solid red curve in Fig. 8A). In contrast, the model predicted that it would have taken 19 s to complete the test if the participant used a combination of a small TS, small WM, and PP of 1 (dashed black line in Fig. 8A). The model further predicted that other combinations of underlying processes would produce intermediate times to complete the test (green, cyan, magenta, purple, and gray lines in Fig. 8A). Taking the distribution of actual total times for our entire participant pool into consideration (Fig. 8B), it is evident that our model accurately covered most of the between-participant variability in total time. The model also performed well at predicting individual performances (Fig. 8C). Overall, the model predicted the total time to complete the test with 91.5% accuracy (average across all subjects).

Fig. 8.

Computational model of visual search. A: model predictions of total time for different combinations of TS, WM, and PP. In the key (top left), the first letter indicates TS value and the second indicates WM value. For TS, S indicates a small value (1–4), L indicates a large value (5–23), and R implies random (24). For WM, S indicates a small value (1–3), L indicates a large value (4–8), and N indicates that it was not used (0). Dashed and solid lines show PP = 1 and PP = 0, respectively. B: histogram of total time for all 69 participants (TMT-A and TMT-B combined). Note that the Y-axes of A and B are identical, showing that predicted values from the model covered almost the entire range of actual total times. C: model predictions (magenta lines) and actual total times (blue lines; Exp. data) for 3 representative participants. D–G: correlations between model parameters and behavioral measures. D: scatter plot of TS vs. search distance. E: scatter plot of TS vs. unique fixations. F: scatter plot and logistic regression of the probability of using WM vs. saccades between refixations. G: scatter plot and logistic regression of the probability of using PP vs. refixations on traversed targets. In D–G, blue circles show individual participants and red lines show linear (D and E) and logistic (F and G) regression fits. In F and G, green circles represent the cumulative probability of using WM (F) or PP (G). For example, in F, if saccades between refixations = 4, then the green circles indicate that the probability that the model predicts a WM > 0 is slightly greater than 0.5. For a good fit in F and G, the red regression lines should overlap the green circles.

In general, TS was larger for TMT-B (6.35 ± 0.57) than for TMT-A (4.49 ± 0.25) and was also larger for stroke survivors (8.53 ± 1.05) than for young (4.1 ± 0.21) and older controls (5.34 ± 0.38). The main effects of group and task (P < 0.001) were significant, and the group × task interaction (P < 0.01) was significant. WM was smaller for TMT-B (0.70 ± 0.2) than for TMT-A (1.54 ± 0.22). Similarly, WM was smaller for the stroke survivors (0.44 ± 0.26) than for young (1.38 ± 0.21) and older controls (1.19 ± 0.32). However, our statistical model revealed that only the main effect of task (P < 0.01) was significant. Our model also showed that in TMT-A, 30 young controls, 13 older controls, and 6 stroke survivors used peripheral visual processing (PP = 1). In TMT-B, these numbers were 13, 3, and 2, respectively. We mentioned earlier that controls and stroke survivors made many saccades back to traversed targets, and yet our model showed that a reasonable number of people used peripheral visual processing. We recognize that the binary stratification of PP may not have captured the complexity of the information measured by fixations of traversed targets. However, creating a continuous model for PP is complicated and outside the scope of the current report.

TS was strongly correlated with the two behavioral measures of spatial planning, search distance and unique fixations (Fig. 8, D and E). This confirms that TS is a reasonable approximation for spatial planning. On the basis of the model prediction for WM, we divided our participants into two groups: those who used working memory (WM > 0) and those who did not (WM = 0). A logistic regression between saccades between refixations and working memory usage provided an excellent fit to the data (Fig. 8F). In contrast, a logistic regression between fixations of traversed targets and PP (0 or 1) did not provide a good fit (Fig. 8G). These results suggest that WM represents saccades between refixations reasonably well but that PP did not adequately model fixations of traversed targets.

To test how the underlying processes of visual search contributed to task performance on TMT, we performed a cluster analysis on TS, WM, PP, and total time (kmeans function in MATLAB). First, we found that including or excluding PP from the analysis did not change the individual clusters. Given that PP also failed to adequately model the experimental data (Fig. 8G), we removed PP from further analysis. Second, we did not observe any qualitative differences in the clusters when TMT-A and TMT-B were analyzed separately. Thus we combined the data from TMT-A and TMT-B (TS, WM, and PP values were calculated separately from both tasks and then pooled together).

Using only TS, WM, and total time, we subsequently found three main clusters (Fig. 9A) that did not change meaningfully if we increased the number of clusters beyond three. The red cluster, which included mostly controls, did not use working memory (WM = 0) but used spatial planning that was restricted to relatively small search spans (TS ≤ 3; Fig. 9B). The green cluster, which also consisted mostly of controls, used working memory (2 ≤ WM ≤ 6) together with spatial planning that comprised broader search spans (3 ≤ TS ≤ 11; Fig. 9C). Interestingly, task performance of the two groups did not differ statistically (total times: 29.47 ± 1.54 and 29.51 ± 1.12 s for green and red clusters, respectively, P > 0.05; Fig. 9, B and C). In contrast, the blue cluster, which consisted mostly of stroke survivors, did not use working memory (WM = 0) but used spatial planning comprising relatively large search spans (3 ≤ TS ≤ 24; Fig. 9, B and D). This group exhibited much poorer performance on TMT (total times: 65.16 ± 6.07 s).

Fig. 9.

Cluster analysis of model parameters and total time. A: TS and WM vs. log of total time (TMT-A and TMT-B combined; both control groups combined). B: TS vs. log of total time. C: WM vs. log of total time. D: TS vs. WM. Green cluster comprises mostly controls and 3 stroke survivors who used working memory (2 ≤ WM ≤ 6) combined with a larger TS for spatial planning (3 ≤ TS ≤ 11). Red cluster comprises mostly controls, who did not use working memory (WM = 0) and performed visual search using a smaller TS for spatial planning (TS ≤ 3). Blue cluster comprises mostly stroke survivors who did not use working memory (WM = 0) and performed visual search using a larger TS for spatial planning (3 ≤ TS ≤ 24).

A strong correlation (ρ = 0.75, P < 0.001) between TS and log(total time) and a weak correlation (ρ = −0.22, P < 0.01) between WM and log(total time) suggests that spatial planning of visual search had a stronger effect on task performance than working memory. Although working memory did not significantly impact task performance when a small search span was used, it had an impact on performance when individuals used a larger search span.

DISCUSSION

The first objective of the study was to investigate how deficits in visual processing, oculomotor control, and cognitive functions contribute to poststroke impairments of visual search. Our prediction was that stroke would affect spatial planning, working memory, and peripheral visual processing simultaneously and that these deficits would contribute to poor task performance. Our results confirmed the presence of stroke-induced deficits in spatial planning and working memory. Most of our participants (both controls and stroke survivors) made many refixations on traversed targets, suggesting that peripheral visual processing (as defined in this study) did not play an important role in the visual search. We also predicted that when the cognitive demands of the task were high (TMT-B), participants would minimize the use of the three search processes and conduct visual search in a more unstructured fashion. We expected this effect to be stronger in the stroke survivors. Our results confirmed these last two predictions.

Our second objective was to develop a task-specific model of visual search to examine how spatial planning, working memory, and peripheral visual processing contribute to visual search. Our model showed that stroke survivors exhibited deficits in spatial planning and in integrating it with working memory, and these deficits also contributed to poor performance on the test. However, the experimental measure for peripheral visual processing was not strongly captured by the model, and the cluster analysis revealed that it had a minimal effect on visual search.

Our behavioral data also showed that visual search (number of fixations) was tightly coupled to task performance (total time). We subsequently examined several measures related to perceptual, oculomotor, and cognitive processes involved in visual search. Saccade duration was assumed to reflect oculomotor control, whereas fixation time was assumed to reflect several processes, including visual recognition, decision-making, and motor planning. Many clinical studies have shown that saccade velocities (for the same saccade amplitude) are slower in patients affected by schizophrenia (Fukushima et al. 1990), traumatic brain injuries (Heitger et al. 2009), and Parkinson's disease (Nakamura et al. 1991). Furthermore, slower saccade speeds, saccade reaction times, and inaccurate saccades in these studies have been broadly classified as “oculomotor deficits.” Surprisingly, neither saccade duration nor fixation duration exhibited a main effect of group or task, and they did not exhibit a significant correlation to task performance. Therefore, these data (shown in Fig. 5) eliminate the possibility that slower saccades and longer fixation durations contributed to prolonged visual search time in stroke survivors. This suggests that, following a left MCA stroke, neither perceptual nor oculomotor impairments made defining contributions to abnormal visual search or task performance during TMT. In contrast, one of our measures of cognitive function, spatial planning (search distance, unique fixations), was strongly correlated with number of fixations (ρ > 0.8 for both measures of spatial planning) and was the most important contributor to visual search.

Considerable evidence demonstrates that high-level global information is initially extracted from a scene and is then used to guide visual search to areas where searched items are most likely to be found (Neider and Zelinsky 2006; Wolfe et al. 2011). On this basis, we expected that healthy controls would exhibit superior spatial planning by restricting their visual search to a small area close to their current hand location, which was the most likely location of the next target. We also expected that stroke survivors would exhibit deficits in spatial planning of their visual search. Our data confirmed this and showed that spatial planning had the strongest influence on task performance (Fig. 9B).

Working memory is another cognitive function that may play a role in guiding visual search. However, some studies have shown that working memory contributes to visual search (Peterson et al. 2001; Woodman et al. 2001), whereas others have found that it plays a minimal role in visual search (cf. Horowitz and Wolfe 1998). We expected that working memory would help guide visual search back to a recently viewed target when it was being searched and would inhibit visual search from returning back to recently fixated targets that were not being searched. Our behavioral measure of working memory (saccades between refixations) showed that many controls used working memory to guide their visual search during TMT-A but not during TMT-B. Furthermore, our computational model showed almost an even split in the number of participants who used working memory and those who did not. These observations suggest that working memory can be used to guide visual search, but only in some individuals when the cognitive demands of a task are relatively low. The computational model also revealed that only three stroke survivors used working memory in TMT-A. These participants had perfect scores on the ViCA, our clinical assessment of cognition (Table 1; scores on individual tests not shown). Surprisingly, two other stroke survivors who had perfect ViCA scores did not use working memory.

A key strength of our computational model is that it enabled us to investigate how different underlying processes of visual search interact combinatorially with each other to guide visual search. Control participants manipulated spatial planning based on their use of working memory. If they did not use working memory, they searched within a smaller topographic area (Fig. 9, red cluster). If they used working memory, they searched within a larger topographic area (Fig. 9, green cluster). In contrast, stroke survivors were typically unable to integrate spatial planning and working memory (Fig. 9, blue cluster). Instead, they generally did not use working memory and searched within a larger topographic area, resulting in poor task performance.

Many studies have examined the relationships between eye movements and cognitive processes during reading (Rayner 1998; Wolfe 1994b) and scene perception (Henderson 2003; Neider and Zelinsky 2006). Eye movements are particularly important for directing overt attention during visual search. During overt attention, objects of interest are attended to with direct fixation. In contrast, covert attention involves attending to objects through peripheral vision (Carrasco and Yeshurun 1998; Posner et al. 1987). Findlay and Gilchrist also showed that during unconstrained visual search, peripheral processing of visual information is minimized and most information is acquired by directly fixating objects of interest (Findlay and Gilchrist 1998). This may partially explain why most of our participants did not use peripheral vision to search targets among traversed targets. An alternative possibility is that the novelty of the task prevented peripheral processing of visual information. It has been suggested that practice improves peripheral information processing (Summala et al. 1996), and it is very likely that our participants would have done the same had they performed multiple trials. However, as stated in materials and methods, our participants performed only one trial because that is how the test is implemented in the clinic and it is most sensitive to detecting neuropsychological deficits in clinical populations. Within the one-trial framework, peripheral visual processing perhaps played a minimal role during unconstrained visual search.

Consistent with our prediction, our computational model revealed that participants employed well-organized visual search in TMT-A but searched for targets in a more unstructured fashion in TMT-B, and this led to a higher number of saccades in TMT-B. This indicates that any change in the cognitive demands of a task may cause concomitant declines in the ability to use cognitive/perceptual processes to gather task-relevant information. In stroke survivors this effect was further magnified, because they made more fixations and performed worse than controls in TMT-B (longer times to complete TMT-B). This implies that any stroke-induced declines in information processing capacity may negatively influence visual search and visuomotor performance concomitantly.

One limitation of the model is that the topographic search span (TS) has been simplified as a vector with ordinal neighbors rather than the actual spatial neighbors. This was done for computational simplicity because most ordinal neighbors were also spatial neighbors. We preferred a simpler model for TS because the focus of the current report was on creating a reasonable mathematical description of visual search deficits in a clinical population. In the future, we plan to implement and test a search model that replicates the spatial topography of the targets.

To conclude, we examined how three underlying processes of visual search (spatial planning, working memory, and peripheral visual processing) contributed to normal and impaired visual search during a task that requires visual search to guide motor performance. We show that deficits integrating spatial planning and working memory underlie abnormal visual search in stroke survivors. By combining our computational model with the Trail Making Test and eye tracking, we were able to show how stroke-induced deficits in cognitive processes affect visuomotor performance. This is important because it provides evidence that stroke-induced impairments of visual search may contribute to difficulties performing daily tasks such as dressing, walking, and driving. Our method can be used to model visual search deficits in other clinical populations that suffer from simultaneous impairments of visual search and motor performance, such as traumatic brain injury (Hills and Geldmacher 1998), Parkinson's disease (Archibald et al. 2013), and Alzheimer disease (Parasuraman et al. 2000). Furthermore, our model can be easily adapted to investigate how different cognitive and perceptual processes interact combinatorially in other similar clinical visual search tests (Mapstone et al. 2003; Trenerry 1990).

GRANTS

The research was partially supported by ASPIRE Grants from the University of South Carolina and National Institute of General Medical Sciences Grant T32 GM081740.

DISCLOSURES

No conflicts of interest, financial or otherwise, are declared by the authors.

AUTHOR CONTRIBUTIONS

T.S., C.M.P., and A.R. performed experiments; T.S. and S.C.T. analyzed data; T.S. and T.M.H. interpreted results of experiments; T.S. prepared figures; T.S., C.M.P., and T.M.H. drafted manuscript; T.S., J.F., C.M.P., S.C.T., S.F., and T.M.H. edited and revised manuscript; T.S., J.F., C.M.P., S.C.T., S.F., and T.M.H. approved final version of manuscript; J.F., S.F., and T.M.H. conception and design of research.