Abstract
How are second-order (texture-defined) and third-order (pattern-tracking) motions processed in our brains? As shown here in the context of an ambiguous motion task involving a nominal second-order stimuli first devised by Werkhoven et al., [Werkhoven, P., Sperling, G. & Chubb, C. (1993) Vision Res. 33, 463–485.], the observers fell into two distinct groups based on the direction of perceived motion. The differences were interpreted in terms of the algorithms used to extract motion: one group by using a second-order motion process and the other by using a third-order motion process. This was investigated further using a dual-task paradigm in which the interference between two tasks indicated the nature of processing involved. Observers who used third-order motion processing experienced interference with letter recognition and a more severe interference in dual third-order motion tasks. Observers who used second-order motion processing experienced interference with another second-order motion detection but not with letter recognition. Insofar as task interference implies the need for attention, the complex interference effects and the apparently paradoxical interference effects of second-order motion perception imply that there are multiple forms of attention. Whether two tasks interfere depends on whether they require the same form of attention. Insofar as spatio-temporal processing is assumed to be carried out in the dorsal stream and pattern recognition in the ventral stream, the interference patterns suggest that second-order motion may be computed entirely in the dorsal stream, and third-order motion may involve two computational processes, one of which shares computational resources with the letter recognition task in the ventral stream.
Imagine the car of the future on the six-lane freeways of Los Angeles, CA: As we drive, a head-up display of the road map hovers in the near field of our vision. While we are paying attention to figure out the next exit shown on the road map, will our brains still be able to do motion processing to avoid the car swerving into our lane in front of us? Splitting attention always has been an interesting subject. How well can we do two tasks together? Can we play the piano and sing at the same time without degradation of either task? Motion detection and shape recognition in primates are known to be processed in the dorsal and ventral streams, respectively (1–7). Yet, these two streams are not entirely separate entities. Cross-talk is found between these two streams (8–11). In this study, an experimental paradigm with dual tasks of motion detection and object recognition was used. The aim was to test whether these two tasks, which are computed in two different streams, would interfere with each other.
Recognition tasks easily can be made arbitrarily difficult. One difficult motion task involves an ambiguous second-order motion stimulus (Fig. 1) devised by Werkhoven, Sperling, and Chubb (12). The stimuli in their study consisted of ambiguous second-order motion displays in which apparent motion is carried by the textural properties. In these stimuli, identical displays can be perceived to rotate in either clockwise or counter-clockwise directions. A heterogeneous motion direction, defined by alternating sectors of texture s (standard) and texture v (variable), competes with a homogeneous motion direction, defined solely by sectors of texture s (12–14). In Fig. 1, the heterogeneous direction is clockwise, and the homogeneous direction is counter-clockwise. Werkhoven et al. (12) found that the human visual system can perceive motion along either the heterogeneous or the homogeneous direction (see Fig. 1), depending on the respective contrast and spatial frequencies of the two textures s and v. I used this stimulus as the motion stimulus for the experiments reported here.
The dual tasks paradigm is a standard tool for studying divided attention (15, 16). Sperling (15) devised a useful tool, the attention operating characteristic (AOC) graph, in which performance of one task is plotted against the performance of another to determine whether two tasks interfere with each other or if they can operate concurrently. When the dual task performance lies on the upper right corner, the two tasks do not interfere. If it lies between the single task performances, the two tasks interfere.
METHODS AND RESULTS
Four sets of experiments were performed. Experiment I involved a single task identical to that in Werkhoven et al. (12). Experiments II–IV involved dual tasks. In Experiment II, observers simultaneously performed a rapid serial visual presentation (RSVP) letter identification task (15, 17) and the apparent motion task. In Experiment III, observers performed dual motion tasks. Experiment IV was a control in which two letter recognition tasks were performed (18, 19).
The procedure of Experiment I is the same as in Werkhoven et al. (12). Fig. 1 schematically illustrates how competing (homogeneous vs. heterogeneous) motion directions are constructed. Fig. 2 shows the actual stimuli and results. Observers had to fixate at the central fixation point. From trial to trial, the direction of rotation and the contrast of texture v were chosen randomly. Observers had to perform a forced-choice task, reporting whether the direction of motion is clockwise or counter-clockwise. The observers in the current study fell into two categories. For 6 of 10 observers, the percentage of trials in which the heterogeneous direction of motion was perceived increased sigmoidally as the contrast modulation mv of texture v increased (Fig. 2, Group A). This was in accordance with the results of Werkhoven et al. (12). However, 4 of 10 observers perceived mostly homogeneous motion for all values of mv, as shown in Fig. 2, Group B. They perceived the direction of motion of the sectors with texture s without explicit instructions to do so. Observers reported that they had fixated at the center. When asked afterward about their subjective perception, they reported having perceived four sectors with texture s. This ruled out the possibility of tracking one particular sector. Instead, these observers seemed to perceive motion carried by the attended pattern. The performances of most observers lay at one of the two extreme ends of the distribution of heterogeneous motion. This led me to suspect that the perception of the ambiguous second-order motion stimuli also may be processed through another pathway that computes motion of particular patterns.
Because of the bimodal grouping of observers, in Experiments II and III, observers were split into two groups; each group was given different instructions. The observers in group A, whose responses in experiment I were similar to those in Werkhoven et al. (12), were asked to perceive global motion while fixating at the center. The observers in Group B were asked to perceive motion of sectors with texture s, i.e., to attend selectively to s while fixating at the center. Group B observers attended to the higher spatial frequency texture s, so they were assumed to have activated the third-order (pattern-tracking) motion mechanism (20, 21).
The two motion detection intructions were intended to amplify the natural tendency of the observers. In preliminary experiments in which no explicit intructions were given, the results were qualitatively similar. On the other hand, when subjects were asked to do what the opposite group did, their performances, even on single tasks, often fell to chance, making it impossible to investigate the dual tasks effect.
For each combination of dual tasks, five attentional instructions were given: two single task instructions and three dual task instructions, namely dual tasks with attention given to task 1, dual tasks with attention given to task 2, and dual tasks with attention equally split between tasks 1 and 2. Blocks of trials gathered under different instruction were treated as separate data points on the AOC graph. The three dual-task instructions served to spread out the data in the AOC graphs.
In Experiment II, observers were required to concurrently perform a RSVP letter identification task (5, 7) as well as the apparent motion task. RSVP letter task: A target letter (a vowel) is presented before the annular motion stimulus is shown. Eight letters, subtending 0.5° of visual angle, are flashed in sequence together with the motion stimulus. Each frame lasts for 115-130 ms, depending on each observer’s performance on letter recognition task with respect to frame time. The target letter appears between 0 and 3 times in this sequence; consonants serve as distractors. The task of the subject is to count the number of times (0–3) the target letter appeared in the sequence. Two precautions are used to avoid repetition blindness (22): No target letters are allowed to appear in two successive frames, and a long time delay of 533 ms separates the appearance of the cue that indicates the target letter and the appearance of the first of the eight letters in sequence. Fig. 3 shows the performance of the observers. First consider group A. The dual task results lie on the upper-right corner of the AOC curves (Fig. 3, Group A). This indicates no interference between the two tasks. For group B, interference occurred between the RSVP letter recognition and third-order motion detection. Fig. 3, Group B shows the data and linear fit for the dual task performance.
In Experiment III, two ambiguous motion stimuli were put together in the form of inner and outer annuli for dual motion perception tasks. Group A was asked to perceive global motion, and group B was asked to perceive motion of the sectors with texture s. Results are shown in Fig. 4. Interference occurred for both group A and group B observers, but the patterns of interference were different. For group A, the experimental dual task performance pattern was straight or slightly curved line segments above the diagonal line joining the single task performance. For group B, the dual third-order motion task performance data scattered in a concave region under the diagonal line joining the single tasks performance in the AOC graph, as shown in Fig. 4, group B. Control experiments of dual letter recognition tasks (18, 19) also were performed. Interference was found between two letter recognition tasks. The result confirms the previous finding (16) that two letter recognition tasks interfere with each other.
The results of the experiments reveal fundamental differences in the way the ambiguous motion stimulus of Figs. 1 and 2 was perceived by group A and group B observers. If we assume that there is only one pathway and interpret the data in Experiment II to mean that, when subjects attended to the central RSVP task, the saliency of higher spatial frequency sectors decreased, then both groups A and B should have similar patterns in the AOC graphs for Experiment III dual motion tasks, either straight lines or curves. This idea contradicts the results in Experiment III that show different patterns in group A’s vs. group B’s AOC graphs. A more possible reason is that the ambiguous motion stimulus is processed by two motion pathways in the brains, a second-order motion pathway and a third-order (pattern-tracking) motion (20, 21) pathway. When observers attended to a particular pattern, such as texture s, motion comprised only of this pattern (the homogeneous direction) was seen by the third-order system. The second-order pathway computed the motion to be in either the homogeneous or the heterogeneous direction, depending on the activities of the sectors (12). These two pathways are different in their architectures (discussed in more detail in Discussion), giving rise to the different patterns of AOC graphs in the experiments in this study.
In summary, two pathways were found for the ambiguous motion stimuli, a second-order motion pathway and a third-order motion pathway. Interference occurred for dual letter recognition tasks. A third-order motion task interfered with an RSVP letter task, and it interfered with another third-order motion task in a more severe manner. A second-order motion task interfered with another second-order motion task, but a second-order motion task did not interfere with an RSVP letter task.
DISCUSSION
Model.
Attention generally has been assumed to be related to the fraction of limited resources one uses (23). In the brain, there are certain areas that provide the bottleneck resources for the computations necessary to perform each task. Here, the computational bottleneck resources will be termed “processors,” and the pathways of these processors will be termed “pathways.” Using a model of limited resources, one can derive quantitatively the different shapes of AOC graphs from the assumed architecture of the processors (see Appendix).
Under this model, all tasks require attention. It is only a matter of how much attention the task requires. The level of interference can be simulated with the equations in the Appendix. The only time two tasks do not interfere is when their computational pathways are nonoverlapping. Different pathways give rise to different forms of attentions.
Two letter recognition tasks required the same form of attention, as did two second-order motion detection tasks and two third-order motion detection tasks. Therefore, interference occurred in all of these dual task combinations. However, the second-order motion detection task required a different form of attention from the RSVP letter recognition task. Hence, no interference occurred in this combination. On the other hand, part of the third-order motion computation required the same form of attention as the RSVP letter recognition task did, so interference occurred (18, 19).
The results of all of the four experiments in this study can be captured by the simple model architecture in Fig. 5 with two computational pathways (Appendix). The first computational pathway performs third-order motion detection and consists of a pattern recognition processor (PRP) that performs pattern recognition functions, followed by a motion-from-pattern processor (MPP) that performs motion computations on attended patterns of the stimulus. Second-order motion detection occurs along a separate computational pathway, which consists of the second-order motion processor (SMP). PRP is separated from SMP. MPP may or may not overlap with SMP, which is outside the scope of this study.
The different perceptions of motion by group A and group B observers in Experiment I can be interpreted as: Group A observers primarily used the second-order motion pathway, and group B observers primarily used the third-order motion pathway. In Experiment II, RSVP letter task and third-order motion detection both required resources from the PRP, leading to interference in group B observers. Second-order motion detection occurred in SMP, which was separate from PRP, explaining the lack of interference between the RSVP letter recognition task and second-order motion detection for group A in Experiment II. In fact, for some group A observers, the RSVP letter task even improved their second-order motion perception because they may have used some third-order motion mechanism when they did motion detection only. Note that, in the AOC graphs of Fig. 3, Group B, the motion direction judgement for the dual task fell to chance before the letter recognition performance reaches single task performance; when observers were paying attention to the RSVP letter task, their performance on the third-order motion detection was poor, hence the motion perception was given by the second-order mechanism. In Experiment III, the concave nature of the AOC graphs for dual third-order motion tasks further supports the hypothesis that the two processors PRP and MPP are separate, consecutive stages in third-order motion detection (Appendix).
The Model in Relation to Physiological Findings.
In the primate visual system, it is believed that shape recognition is processed through the ventral, or “what” stream, which passes through the inferior temporal lobe, and motion is processed through the dorsal, or “where” stream, passing through the medial temporal lobe (24–27) and medial superior temporal lobe (28). According to these neurophysiological findings, the RSVP letter task probably is processed in the ventral stream, somewhere near the human equivalent of the inferior temporal lobe. Because third-order motion shares the PRP with the RSVP letter task, PRP also is located in the ventral stream. The MPP of the third-order motion pathway, being completely distinct from the PRP, is possibly in the dorsal stream. Second-order motion is proposed to be processed through V1-V2-MT (29, 30). Our experiments show that it does not interfere with the RSVP letter recognition task. This result supports the idea that second-order motion processing, and hence the SMP of the model in Fig. 5, does not go through the human equivalent of the inferior temporal lobe, and may be entirely in the dorsal stream. This is consistent with the models of Wilson et al. (29) and the findings of Vaina and Cowey (30).
Acknowledgments
This paper owes much to the inspiration and invaluable guidance of Prof. G. Sperling, who has been a great coach and mentor. I also gratefully acknowledge Profs. C. Koch, F. Crick, J. Allman, K. Lau, J. Braun, and S. Shimojo for their constructive comments and support. This research was funded by the Lawrence Hansen Foundation and the Caltech Engineering Research Center and was conducted in Prof. C. Koch’s laboratory.
ABBREVIATIONS
- AOC
attention operating characteristic
- RSVP
rapid serial visual presentation
- PRP
pattern recognition processor
- MMP
motion-from-pattern processor
- SMP
second-order motion processor
Appendix
This appendix presents equations that model performance with resource constraints (see also ref. 23 for related models).
Two Tasks Sharing the Same Resources
Suppose that two tasks, X and Y, share the same resource. Let the fraction of resource used for task X be β. A simple assumption for the performance Px and Py is a piecewise linear function in β as follows:
1 |
2 |
where a and c are chance probabilities, b and d are maximum performance, and xt and yt are the transition points, at which the performances reach their maximum values, for tasks X and Y. In other words, xt and yt are the fractions of resources needed to reach the maximum performances of tasks X and Y. If both xt and yt are equal to 1, the dual task performance will be on the diagonal line joining single task performance. If both xt and yt are 0, the dual tasks performance will be on the upper right corner. For cases when xt and yt are between 0 and 1, it will be a straight line above the diagonal.
Two Tasks Sharing the Same Compound Attention Processor
If the processor has two stages that are used in conjunction, Px depends on α and β, the fractional resources used in the two stages. The performance will be a product of these two variables.
3 |
The equations for Py(1 − α, 1 − β) are analogous. The quadratic term leads to a concave area under the diagonal line instead of a straight line on AOC graphs. This explains the curvature found in AOC graphs of the dual pattern motion experiment.
Two Computation Pathways Architecture
The results of my experiments led me to propose the architecture for the second-order motion stimuli and letter recognition tasks as shown in Fig. 5. Two separate computation pathways compute motion: the second-order motion pathway consisting of SMP and the third-order motion pathway consisting of PRP and MPP. Inputs are sent to both pathways. Each pathway computes the given tasks independently, and the results compete according to their outputs and their weights. Assuming the decision process is a summation of inputs from the two pathways and for simplicity, assuming PRP and MPP have the same utilization on average, one has:
4 |
5 |
where Px and Py are the final performances of tasks X and Y, respectively. PRP and MPP are assumed to have the same utilization α1 whereas α2 is the utilization for SMP. Px and Py are analogous performances for the computation pathway along PRP and MPP. Qx and Qy are performances for the computation pathway along SMP for tasks X and Y, respectively. q is the normalized weight for the output of the computation pathway along PRP and MPP. This weight is close to 0 for group A observers and close to 1 for group B observers. It can be changed voluntarily when one wants to pay attention to a certain pattern. q is small when perceiving global motion, and q is close to 1 when perceiving the motion of certain patterns.
References
- 1.Pohl W. J Comp Physiol Psychol. 1973;82:227–239. doi: 10.1037/h0033922. [DOI] [PubMed] [Google Scholar]
- 2.Ramachandran V-S, Gregory R-L. Nature (London) 1978;275:55–56. doi: 10.1038/275055a0. [DOI] [PubMed] [Google Scholar]
- 3.Ungerleider L-G, Mishkin M. In: Analysis of Visual Behavior. Ingle D-J, Goodale M-A, Mansfield R-J-W, editors. Cambridge, MA: MIT Press; 1982. pp. 549–586. [Google Scholar]
- 4.Livingstone M, Hubel D H. J Neurosci. 1987;7:3416–3468. doi: 10.1523/JNEUROSCI.07-11-03416.1987. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.DeYoe E A, Van Essen D C. Trends Neurosci. 1988;11:219–226. doi: 10.1016/0166-2236(88)90130-0. [DOI] [PubMed] [Google Scholar]
- 6.Shipp S, Zeki S. Nature (London) 1988;335:311. doi: 10.1038/335311a0. [DOI] [PubMed] [Google Scholar]
- 7.Zeki S. In: Vision: Coding and Efficiency. Blakemore C, editor. Cambridge, U.K.: Cambridge Univ. Press; 1990. pp. 321–345. [Google Scholar]
- 8.Blasdel G-G, Lund J-S, Fitzpatrick D. J Neurosci. 1985;5:3350–3369. doi: 10.1523/JNEUROSCI.05-12-03350.1985. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Rockland K-S. J Comp Neurol. 1985;235:467–478. doi: 10.1002/cne.902350405. [DOI] [PubMed] [Google Scholar]
- 10.Shipp S, Zeki S. J Neurosci. 1989;1:309–332. doi: 10.1111/j.1460-9568.1989.tb00798.x. [DOI] [PubMed] [Google Scholar]
- 11.Shipp S, Zeki S. J Neurosci. 1989;1:333–354. doi: 10.1111/j.1460-9568.1989.tb00799.x. [DOI] [PubMed] [Google Scholar]
- 12.Werkhoven P, Sperling G, Chubb C. Vision Res. 1993;33:463–485. doi: 10.1016/0042-6989(93)90253-s. [DOI] [PubMed] [Google Scholar]
- 13.Werkhoven P, Snippe H, Koenderink J. Percept Psychophys. 1990;47:509–525. doi: 10.3758/bf03203104. [DOI] [PubMed] [Google Scholar]
- 14.Werkhoven P, Koenderink J. J Opt Soc Am A. 1991;8:1510–1516. doi: 10.1364/josaa.8.001510. [DOI] [PubMed] [Google Scholar]
- 15.Sperling G, Melchner M-J. Science. 1978;202:315–318. doi: 10.1126/science.694536. [DOI] [PubMed] [Google Scholar]
- 16.Duncan J. Percept Psychophys. 1993;54:425–430. doi: 10.3758/bf03211764. [DOI] [PubMed] [Google Scholar]
- 17.Sperling G, Budiansky J, Spivak J-G, Johnson M-C. Science. 1971;174:307–311. doi: 10.1126/science.174.4006.307. [DOI] [PubMed] [Google Scholar]
- 18.Ho E-C, Koch C. Invest Opthalmol Visual Sci. 1996;37:3. [Google Scholar]
- 19.Ho, C.-E. (1996) CNS Technical Report CNS-TR-96-02.
- 20.Lu Z-L, Sperling G. Nature (London) 1995;377:237. doi: 10.1038/377237a0. [DOI] [PubMed] [Google Scholar]
- 21.Lu Z-L, Sperling G. Vis Res. 1995;35:2697. doi: 10.1016/0042-6989(95)00025-u. [DOI] [PubMed] [Google Scholar]
- 22.Kanwisher N-G, Yin C, Wojciulik E. In: Fleeting Memories. Coltheart V, editor. Cambridge, MA: MIT Press; 1997. , in press. [Google Scholar]
- 23.Sperling G, Dosher B-A. In: Handbook of Perception and human performance. Vol I. Boff K-R, Kaufman L, Thomas J-P, editors. New York: Wiley; 1986. pp. 2.1–2.65. [Google Scholar]
- 24.Maunsell J, Van Essen D-C. J Neurophysiol. 1983;49:1127–1147. doi: 10.1152/jn.1983.49.5.1127. [DOI] [PubMed] [Google Scholar]
- 25.Maunsell J, Van Essen D-C. J Neurophysiol. 1983;49:1148–1167. doi: 10.1152/jn.1983.49.5.1148. [DOI] [PubMed] [Google Scholar]
- 26.Albright T. Science. 1982;255:1141–1143. doi: 10.1126/science.1546317. [DOI] [PubMed] [Google Scholar]
- 27.Albright T. J Neurophysiol. 1984;52:1106–1130. doi: 10.1152/jn.1984.52.6.1106. [DOI] [PubMed] [Google Scholar]
- 28.Graziano M-S-A, Andersen R-A, Snowden R-J. J Neurosci. 1994;14:54–67. doi: 10.1523/JNEUROSCI.14-01-00054.1994. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Wilson H-R, Ferrera V-P, Yo C. Vis Neurosci. 1992;9:79–97. doi: 10.1017/s0952523800006386. [DOI] [PubMed] [Google Scholar]
- 30.Vaina L-M, Cowey A. Proc R Soc B. 1996;263:1225–1232. doi: 10.1098/rspb.1996.0180. [DOI] [PubMed] [Google Scholar]