Current directions in visual perceptual learning

Abstract

The visual expertise of adult humans is jointly determined by evolution, visual development, and visual perceptual learning. Perceptual learning refers to performance improvements in perceptual tasks after practice or training in the task. It occurs in almost all visual tasks, ranging from simple feature detection to complex scene analysis. In this Review, we focus on key behavioral aspects of visual perceptual learning. We begin by describing visual perceptual learning tasks and manipulations that influence the magnitude of learning, and then discuss specificity of learning. Next, we present theories and computational models of learning and specificity. We then review applications of visual perceptual learning in visual rehabilitation. Finally, we summarize the general principles of visual perceptual learning, discuss the tension between plasticity and stability, and conclude with new research directions.

TOC blurb

Perceptual learning, or performance improvements after training on perceptual tasks, is a widespread phenomenon in visual perception. In this Review, Lu and Dosher describe findings of specificity and transfer of perceptual learning, mechanisms of learning, and key applications in visual rehabilitation

Introduction

Perceptual learning, or performance improvements in perceptual tasks through practice or training, has fascinated sensory psychologists from the beginning of experimental psychology.^1,2 Perceptual learning occurs in all sensory modalities, including vision, audition, touch, smell, taste, and multimodal combinations,³ and can significantly improve performance⁴ and persist for years.⁵ Visual perceptual learning in laboratory experiments can improve accuracy in judgements of orientation, spatial frequency, and motion direction from slightly above chance to 90% correct or more,^6–8 increase contrast sensitivity by more than 150%,^9,10 and reduce response times in judgement of motion direction by about 40%.¹¹ Because perceptual performance can be so significantly altered by perceptual learning, a complete understanding of perception requires an understanding of how it is modified by perceptual learning.³

Perceptual expertise also arises across many years of training in real-world tasks.^12–15 For instance, chess masters can perceive and remember meaningful arrangements of pieces on the board, enabling them to plan useful moves better than novices.¹² Expert weather forecasters can extract and classify systematic patterns of visual and other features in satellite images.¹³ The powerful impact of perceptual learning on human performance suggests that perceptual systems are remarkably plastic even in adults whose cortical organization and function are mature.

In real-world tasks, perceptual judgements often involve complex stimuli and performance contexts, and perceptual expertise can be achieved through coordinated improvements of multiple processes.¹³ One approach to understand the basis of perceptual expertise is studying how perceptual learning improves one or more component processes. Thus, studies on perceptual learning in the laboratory typically use relatively simple stimuli and judgments and controlled training protocols.^3,16–21 The goals are to characterize the basic functions, properties, and mechanisms of perceptual learning in each component process, and then apply the principles to understand and facilitate the development of perceptual expertise in the real world, including rehabilitation procedures for patients with sensory deficits.^22–25

A trademark characteristic of perceptual learning in the laboratory is its specificity: performance improvements from training in one task with a particular stimulus and judgment typically fail to transfer to improvements with highly related tasks and stimuli.^3,4,16 In visual perceptual learning, specificity has been reported for stimulus orientation, spatial frequency, motion direction, stimulus pattern, and even location in the visual field. Specificity distinguishes perceptual learning from cognitive learning, strategy selection, and motor learning, which exhibit broader generalizability. However, specificity is a graded phenomenon; there can be some specificity but also some transfer of trained improvements to other stimuli and tasks. The extent of specificity or transfer can be affected by many experimental factors, and the interpretation of specificity in the context of a brain network helps locate the site of plasticity.^3,26 Specificity might hinder practical applications of perceptual learning because it somewhat limits the benefits of training to the trained stimulus and task.¹⁰ The optimization of transfer to untrained stimuli and tasks is therefore an important precursor for applications of perceptual learning in the development of real-world expertise and sensory rehabilitation.²⁷

In this Review, we provide an integrated treatment of key behavioral aspects of visual perceptual learning. We begin with a survey of the visual tasks that exhibit perceptual learning and manipulations that impact the magnitude of learning. We then turn to the trademark phenomenon of specificity. Next, we focus on how perceptual learning changes visual processing. We detail a set of modeling and experimental investigations that enhance the development of new theoretical frameworks and enable more efficient practical applications. Then we review applications of perceptual learning in visual rehabilitation. Finally, we summarize the general principles of visual perceptual learning and propose new research directions.

Although there are notable differences between sensory modalities, including the receptors used to register the sensory inputs and the corresponding representations and perceptual processes in the brain, perceptual learning across sensory modalities shares common methodologies, theories, and computational models. Across modalities, performance improves through training and practice, the extent of specificity and transfer depends on the stimulus and task, and experience-dependent improvements of the signal-to-noise ratio limit perceptual judgments. Thus, a review of visual perceptual learning can also benefit researchers studying other sensory modalities. For specific information about perceptual learning in other modalities, interested readers can refer to specialized review articles.^3,17,28–31

Visual perceptual learning tasks

A wide range of tasks exhibit visual perceptual learning, including simple tasks such as feature detection (‘Is there a horizontal line present?’) and feature discrimination (‘Which of these two lines tilts more clockwise?’) and more complex tasks such as scene analysis and perceptual aspects of sports. Perceptual learning occurs far more often than not in many thousands of studies.^{3,16,17,19,20} Only a few stimuli and tasks have resulted in relatively small or no effects of perceptual learning, mostly involving basic discriminations of orientations or motion direction around the horizontal or vertical meridian in the fovea [G].³² In the majority of cases, if initial performance in a perceptual task is not optimal, it can be improved by perceptual learning.

Standard paradigm

Most visual perceptual learning studies use forced-choice tasks. For example, in a two-interval forced-choice detection task, a visual stimulus and a blank screen are shown to the observer in two separate time intervals within a trial. The observer is required to indicate which interval contains the stimulus. In an N-alternative forced-choice identification task, one of several visual stimuli is chosen randomly and presented to the observer in each trial, who is required to identify the stimulus (Fig. 1a). Usually, some form of feedback follows, either indicating if the response was correct or showing the correct response.

Figure 1.

Standard paradigm. (a) In a perceptual learning task, one of several potential stimuli (marked with a blue circle) is randomly chosen and shown to the observer, who identifies the stimulus (“O3”) and receives either no feedback, feedback regarding their accuracy (right or wrong), or is shown the correct response. (b) In the constant stimuli procedure, stimuli are kept constant throughout training and performance accuracy is measured over the course of training. (c) In the adaptive training procedure, one stimulus feature (here: contrast) is adaptively adjusted throughout training to keep the performance level constant; contrast threshold is measured over the course of training. (d) In the adaptive training paradigm, the difference between the to-be-discriminated stimuli (here: orientation) is adaptively adjusted throughout training to keep the performance level constant; difference threshold is measured over the course of training.

The stimulus set is typically controlled with one of three procedures. In the constant stimuli procedure, the exact same set of visual stimuli is used during the entire course of training (Fig. 1b). Performance accuracy or response time are typically measured across a block (consisting of dozens of trials) or session (consisting of hundreds of trials), and performance is plotted against the number of training blocks or sessions as the learning curve. As training proceeds, the task becomes easier for the observer, that is, they become more accurate and/or faster. In adaptive training procedures, either the contrast of the stimuli (Fig. 1c) or the difference between the to-be-discriminated stimuli (Fig. 1d) is adjusted based on the observer’s responses to keep performance accuracy constant. The learning curve is based on the contrast or difference threshold, typically measured in blocks or sessions. Although visual perceptual learning in these procedures reduces the contrast or difference threshold at which observers can perform the task, their accuracy and therefore the task difficulty remain the same. Variations of these three basic training procedures include the same set of stimuli targeting multiple performance levels,^26,33 a mixture of several adaptive training sequences each targeting distinct performance accuracy levels, or a sequence of constant stimuli and/or adaptive training episodes.^34,35

Typical block-by-block or session-by-session learning curves used in all three procedures sample performance coarsely and therefore gloss over performance changes at other time scales such as initial rapid learning or overnight performance improvements.^36,37 In addition, the stimuli used to assess performance are the training stimuli in the vast majority of existing studies. The coupling of training and assessment limits the range of training stimuli to those near the threshold and might lead to suboptimal learning in these paradigms.^34,35

Key manipulations

Every visual perceptual learning experiment must specify many experimental details, including the stimuli, perceptual task, target response accuracy (for adaptive methods), feedback, reward, attention, training protocol, and any brain stimulation or pharmaceutical intervention if present. Any one of these factors can influence the magnitude and specificity of visual perceptual learning and their effects should not be taken for granted.

For example, an orientation discrimination task might use Gaussian-windowed sinusoidal gratings (or ‘Gabors’) as the stimuli (Fig. 1a). A Gabor is defined by a large number of variables, including its retinal location, contrast, orientation, and the center spatial frequency and phase of the grating. If the Gabor is masked by external noise, additional parameters need to be specified for the noise. The magnitude of learning can depend systematically on several of these stimulus factors.^{10,16,26,32,38,39} For example, initial performance is worse and there is more learning if the stimulus is in the periphery or displayed with external noise.^32,40

The kind of perceptual judgment can also significantly influence perceptual learning. Perceptual learning and degree of specificity might depend on whether the judgment requires detection or discrimination between two or more than two choices.^41,42 The high degree of specificity between training on detection and discrimination tasks with identical or highly similar stimuli suggest that observers learn different aspects of the visual stimuli in the different tasks.^38,43

Feedback and reward also influence perceptual learning. Learning can occur with block feedback, which only informs the observer of the aggregate performance accuracy over many trials,^44,45 or without any feedback.^44–49 However, trial-by-trial feedback can improve the rate of learning or enable learning in otherwise not learnable tasks.^50,51 Random or false feedback can eliminate learning, and biased feedback that consistently mislabels the accuracy of subject’s trial-by-trial responses can induce subjective decision bias in subsequent tests.^44,52–54 One study showed that block feedback that exaggerated the observer’s response accuracy could improve the rate of learning.⁵⁵ In tasks with more than two alternatives, feedback that indicates the correct answer can lead to better learning than accuracy feedback.⁵⁶ Reward for correct responding can also serve as a form of feedback and support learning.^57,58 High monetary reward,⁵⁹ pharmaceutical intervention that increases acetylcholine level,⁶⁰ and brain stimulation can lead to faster learning.⁶¹

Training at a high accuracy in adaptive training procedures can be beneficial, leading to faster learning, and mixing high-accuracy and low-accuracy training improves learning in the slower low-accuracy condition.^35,51,62 Training schedule, including the total number of training days, sessions, trials per session, and intervals between training blocks and sessions can also influence the magnitude of learning. For instance, there is a minimum number of trials per session for observers to learn⁶³ and too many trials in a session can lead to less learning.⁶⁴

Most visual perceptual learning studies use a single task. Studies that explicitly manipulate attention involve multiple tasks or the same task in more than one spatial location, with observers required to attend to one or more tasks at a time. Spatial attention can enhance or even enable perceptual learning. When there is less or even no performance improvement in the unattended location, the magnitude of learning can be much greater in the attended location.^65,66 Interleaving training trials of a target task with those of another task can also facilitate learning of the target task.⁶⁷ Perceptual learning can also occur for subthreshold stimuli that are irrelevant for the observer’s main task but occur in close temporal proximity with the target stimuli in the main task.^18,68

The details of the training procedures influence trial-by-trial learning and long-term accumulation of learning effects over time. In combination with other factors, they also control the total magnitude of performance improvements. Because any individual experiment typically manipulates one or a very limited number of these factors, finding the optimal combination is a challenge.

Learning at multiple levels

Visual perceptual learning occurs across different primary representations of the training stimuli. Here we consider three levels based on the primary representation of the training stimuli: single-feature tasks related to low-level vision, pattern tasks related to mid-level vision, and objects or natural scene tasks that involve high-level vision.

A vast majority of studies have investigated low-level tasks with judgments based on a single feature of the stimuli, coded in early visual cortical areas. Gabors, which mimic the receptive field of V1 neurons, are used in many studies in two-interval or two-alternative forced-choice tasks (Fig. 2a). Learning in these tasks has been observed in detection,³⁸ contrast discrimination,⁶⁹ color contrast discrimination,⁷⁰ orientation discrimination in no noise⁷¹ or external noise⁹, spatial frequency discrimination,⁷² phase discrimination,⁶ acuity⁷³ and hyperacuity [G] tasks^8,48. Substantial learning, sometimes improving performance by a factor of two or more in sensitivity (d′) or threshold reduction, has been observed after hundreds of trials of training. Relatively few cases have documented no learning; these tasks involve dominant forms of stimulus features such as vertical and horizontal (rather than oblique) orientations and stimulus presentation in the fovea (rather than the periphery).^16,32 The magnitude of learning is often greater when the stimuli are of high than low spatial frequency, presented in high external noise than in no external noise, or when the task is carried out in peripheral vision.^3,10,32,38 Because the representations of the individual features exist in the visual system prior to perceptual learning, accounts of low-level perceptual learning posit that the relevant representations for the task are either enhanced and/or selected and their weights or read-out optimized in perceptual decision (Box 1).³

Figure 2.

Tasks that exhibit visual perceptual learning. (a) Low-level feature tasks: Gabor detection, contrast discrimination, color contrast discrimination, orientation identification, orientation identification in external noise, spatial frequency discrimination, phase discrimination, and hyperacuity. Two to-be discriminated stimuli are shown for each task. (b) Mid-level pattern tasks: compound grating discrimination, texture discrimination, depth discrimination, and motion direction discrimination. (c) High-level object/scene tasks: Contour/shape detection, face recognition, and object recognition.

Box 1. The Integrated Reweighting Theory.

The visual system can be construed as a multi-layer hierarchical network.¹⁹⁰ Visual inputs (see image) are initially represented in multiple feature maps in early visual cortical areas, each of which characterizes local properties of the visual input, including orientation, spatial frequency, phase, color, disparity, and local motion. These feature maps are further processed in each local region. Then features from different maps are integrated to generate feature-invariant representations, and further integrated across spatial locations to generate location invariant representations. All feature maps are directly connected to regions that conduct perceptual decision (only the connections between the location invariant representations to decision are shown in the image). The connections between different layers are bi-directional. There could also be recurrent connections within each feature map.

In the integrated reweighting theory, visual perceptual learning optimizes connection weights between layers of the hierarchical network and between each layer and perceptual decision without altering the initial feature maps. The weights are changed using an augmented Hebbian learning rule,^26,127 based on the co-activation between connected regions and external feedback when available. In addition, the rate of weight change can be affected by attention and reward.³

In this framework, the extent of specificity and transfer depends on the degree of overlap and compatibility of connections for the training and transfer tasks. When stimuli in the training and transfer tasks in different locations (or with different features) are separately represented by location-specific (or feature-specific) regions, reweighting the connections from the representation of the stimuli to decision during training does not change the connections for the transfer task, leading to location (or feature) specificity. When stimuli in the training and transfer tasks in different locations (or with different features) are represented by the same location-invariant (or feature-invariant) region, reweighting the connections from the representation of the stimuli to decision during training changes the connections for the transfer task, leading to positive location (or feature) transfer if the two tasks are compatible and negative location (or feature) transfer if they are incompatible. If training and transfer tasks involve both location-specific (or feature-specific) and location-invariant (or feature-invariant) representations, partial specificity and partial transfer would occur.

Mid-level visual perceptual learning tasks use patterns composed of simple visual features. Although the simple visual features are processed in early visual cortical areas, mid-level visual cortical areas are involved in integrating or segregating them into patterns.⁷⁴ For example, in a same-or-different compound grating judgment task (Fig. 2b),⁷⁵ two component sinewave gratings with different spatial frequencies and orthogonal orientations are superimposed to form a plaid. In each trial, either the same plaid or two that differ in spatial frequency of one of the component gratings are presented in two intervals and the observer is asked to indicate if the two plaids are the same or different. Other mid-level tasks include discriminating between textures⁴ and indicating the dominant motion direction of moving random dots⁷.

Learning has been documented in perceptual judgements in mid-level tasks involving compound grating discrimination,⁷⁵ texture discrimination,⁴ visual search,⁷⁶ depth perception,^77,78 and motion direction or speed discrimination tasks.^7,79,80 Relative to low-level tasks, effects of perceptual learning in mid-level tasks are larger.⁸¹ In addition, larger magnitudes of learning are found in tasks involving non-dominant stimuli such as oblique motion rather than horizontal or vertical motion, or with high speed rather than low speed.⁷⁹ Because the representations of the individual features used to create the patterns exist in the visual system prior to perceptual learning in the mid-level tasks, learning occurs through enhancement and/or selection of the relevant representations at both the feature and pattern levels and optimizing their weights in perceptual decision (Box 1).^3,82

Research on visual perceptual learning in high-level tasks involving objects and natural scenes has only investigated a very small subset of potential stimuli. High-level stimuli are processed by high-level cortical areas such as the Inferior Temporal cortex and prefrontal cortex. Strong visual perceptual learning has been documented in contour perception,⁸³ object identification,^84,85 face perception,⁸⁶ and biological motion (Fig. 2c).⁸⁷ Relative to low-level and mid-level tasks, visual perceptual learning in high-level tasks often requires far fewer trials to reach asymptotic high performance levels, and might reflect creation of new high-level representations of objects or scenes by combining existing visual features.³

Although the primary representation of the training stimuli provides one way to organize the literature, there is considerable overlap in the phenomenology across the levels of perceptual learning as well as substantial variations within each level. Furthermore, the relationship between the primary representation of the training stimuli and perceptual learning might be more complex than currently appreciated. Perceptual learning in a low-level feature task might impact higher levels of visual analysis; learning in a mid-level pattern or high-level object task might also involve learning to enhance and/or select low-level features; and learning at all levels might be involved in improved perceptual decision. In addition, substantial individual differences are often observed in visual perceptual learning.⁸⁸ These differences might reflect individual differences in learning ability and are correlated to personality traits such as IQ, openness to experience, extraversion, and neuroticism scores and basic cognitive functions such as simple reaction time and cognitive flexibility.^89,90 In summary, perceptual learning is a complex process that occurs within a complex set of brain networks and might be the result of plasticity at multiple levels.

Specificity and transfer

A hallmark finding in visual perceptual learning is its specificity to retinal location, eye, stimulus, and/or task. However, specificity is not absolute; there is usually some transfer of learning to related stimuli and tasks, which can depend on many factors. The observation of specificity in behavioral results might be consistent with improved weighting or readout from early cortical areas and might not necessarily imply plasticity in early visual cortical areas.^9,26,40 The extent of specificity and transfer is quantified using the specificity index and varies over many task contexts. Multiple factors influence whether transfer arises and its size.

The specificity index

Three common forms of transfer include location, stimulus, and task transfer. For example, an observer might perform a training task to detect a vertical Gabor in the upper-left visual field (Fig. 3a). The Gabor, either symmetric with a 0 deg phase or asymmetric with a 90 deg phase, appears in half of the trials and the observer must indicate if it is present in each trial. The contrast of the Gabor is adjusted across trials to keep observer’s performance at a constant accuracy level. A hypothetical learning curve of contrast sensitivity across trials would monotonically increase and then plateau after a few hundred trials, resulting in a total magnitude of learning referred to as M_learning (the difference between the initial and final contrast sensitivity) (Fig. 3e).

Figure 3.

Training and transfer. (a) A sample training task: A vertical Gabor detection task in the upper-left visual field. The Gabor, with either a 0 or a 90 deg phase, occurs in 50% of the trials. The observer indicates the presence or absence of the Gabor in each trial. The contrast of the Gabor is adaptively adjusted from trial to trial to keep performance at a constant accuracy level. (b) Location transfer: The same vertical Gabor detection task in the lower-right visual field. (c) Stimulus transfer: A horizontal Gabor detection task in the upper-left visual field. (d) Task transfer: A phase discrimination task in the upper-left visual field in which the observer is asked to indicate the phase (0 or 90 deg) of the Gabor in each trial. (e) The contrast sensitivity learning curve in the training task. M_learning is the total amount of learning. (f) The contrast sensitivity learning curve in the location (b) or stimulus (c) transfer task. M_transfer is the total amount of transfer, representing the benefit from the training task. SI is the specificity index.

To test location transfer, the observer performs the same Gabor detection task in the lower-right visual field (Fig. 3b). To test stimulus transfer, a horizontal rather than a vertical Gabor might be used in the detection task (Fig. 3c). Finally, to test task transfer, the observer might perform a phase discrimination task, judging if the Gabor is symmetric or asymmetric, on the original stimulus (Fig. 3d). A hypothetical learning curve in the location and stimulus transfer tasks is a monotonically increasing curve, but with a higher initial contrast sensitivity M_transfer that reflects the benefit from training in the original task (Fig. 3f). The specificity index is defined as one minus the ratio between M_transfer and M_learning. Complete specificity (specificity index of 1) results if there is no benefit from the training task and complete transfer (specificity index of 0) results if there is full benefit from the training task. For task transfer, an additional control experiment is often used to measure visual perceptual learning in the transfer task without initial training, and the learning curves with and without initial training are compared to derive the specificity index.³

Observations of specificity

Although a few studies have documented complete specificity or complete transfer, most find partial specificity and partial transfer (Table 1). Evidence for retinal location specificity has been found in contrast detection,³⁸ orientation discrimination,⁷¹ texture discrimination,⁴ motion-direction discrimination,⁹¹ depth from random-dot stereograms,⁹² and localization tasks.⁹³ The texture discrimination task^4,94 (but see ⁹⁵) also exhibited eye specificity (monocular training benefits the trained eye more than the untrained eye), although orientation discrimination task did not.^71,96 The sinewave motion direction discrimination task exhibited eye specificity without external noise but eye transfer in high external noise.⁹⁷ Visual perceptual learning in hyperacuity,⁸ complex pattern,⁶ motion-direction,⁷ and depth⁷⁷ judgements exhibited large degrees of orientation specificity. Spatial frequency specificity has been demonstrated in contrast detection^10,38 and complex pattern discrimination tasks.⁶ There was also specificity to the trained objects in object recognition.⁸⁴ However, visual perceptual learning is largely non-specific to viewing distance or spatial scale, suggesting that perceptual learning in those tasks involved scale-invariant visual representations.^32,84 Large degrees of task specificity have been demonstrated between motion detection and motion direction discrimination,⁹¹ between orientation and luminance discriminations on lines,⁴⁵ between bisection and Vernier judgments,^98,99 and between global and local judgments on texture patterns.¹⁰⁰ Although complete specificity has been famously reported in some instances such as retinal location specificity in texture discrimination, complete transfer has been reported in eye and viewing distance specificity. The most common observation is partial specificity and partial transfer, with a specificity index between 0 and 1.

Table 1.

Observations of specificity and transfer in visual perceptual learning

Level	Task	Stimuli	Type of transfer tested	Transfer observed	References
Low	Contrast detection	Sinewave gratings	Spatial frequency	Low to high specificity (depending on spatial frequency difference)	10,38
	Hyperacuity	Pairs of parallel lines	Orientation	Full specificity	8
	Orientation discrimination	Oriented bars of random dots	Retinal location	Low to moderate specificity	71
		Gabors with and without noise	Viewing distance	Full transfer	32
	Orientation and brightness discriminations	Lines	Task	Full specificity	45
	Complex pattern discrimination	Compound sinewave gratings	Spatial frequency	Full specificity	6
Mid	Depth discrimination	Random-line stereograms	Orientation	Moderate specificity	77
Mid	Motion direction discrimination	Moving random dots	Retinal location	High specificity	91
Mid		Moving sinewave gratings	Eye	Moderate specificity	97
Mid		Moving sinewave gratings in noise	Eye	Full transfer	97
Mid		Moving random dots	Orientation	Full specificity	7
Mid	Motion detection and motion direction discrimination	Moving random dots	Task	Full specificity	91
Mid	Bisection and Vernier judgments	Patterns of dots	Task	Full specificity	98, 99
Mid	Global and local texture judgments	Textures of lines	Task	Full specificity (local to global). High specificity (global to local).	100
High	Object recognition	Images of common objects	Object	High specificity	84
High	Object recognition	Images of common objects	Viewing distance	Full transfer	84

Note: Degree of transfer is broadly labeled as low, moderate, high, or full transfer.

Extent of specificity and transfer

The relative degree of specificity and transfer depends on many factors, including task difficulty or precision,^101–105 extent of training,¹⁰⁶ state of adaptation,¹⁰⁷ attention,^108–111 sensory uncertainty of stimulus features,¹¹² and cross training.^34,113

One early study concluded that there was more specificity between difficult training and difficult transfer tasks than between easy training and easy transfer tasks.¹⁰¹ However, the study actually manipulated the difference between the stimuli (precision), rather than task difficulty (the performance accuracy level was kept the same in all the conditions). Using a complete factorial design that evaluated all four possible combinations of low-precision and high-precision training and transfer tasks, a later study found that the degree of retinal location specificity in an orientation identification task depended only on the precision, not the difficulty, of the transfer task. There was more transfer to a low-precision than a high-precision transfer task, independent of the precision of the training task.¹⁰³ The same pattern of results has also been observed in texture and motion discrimination tasks.^104,105 High precision tasks require distinguishing between very similar stimuli with acutely tuned judgments, so it is not surprising that transfer from related tasks is less effective.

The extent of training also influences the degree of specificity, with more training often leading to less transfer and more specificity. Shorter training in texture discrimination⁴⁷ and high-precision orientation-discrimination¹⁰⁶ tasks and distributing the same number of training trials over more days in a hyperacuity task¹¹⁴ led to less orientation specificity than more extensive training. The extent of specificity might also depend on the level of visual adaptation for presented visual features, as illustrated in a study of texture discrimination.¹⁰⁷ When the target lines only occurred in one location and the target, background and mask line orientations remained the same throughout training, a large degree of retinal specificity was observed. However, interspersing frames of the texture display with differently oriented lines (but no target) reduced adaptation and greatly reduced specificity. Endogenous and exogenous spatial attention and feature-based attention also significantly reduced retinal location specificity in orientation discrimination and visual acuity tasks.^108–111 Interestingly, feature-based attention enabled location transfer—reminiscent of its effect across the visual field—but preserved orientation specificity.¹¹⁰

Training of a secondary task in a transfer location can promote transfer of a primary task.^113,115 In one study, a high precision orientation discrimination task with near vertical stimuli was highly specific to retinal location, but training a secondary horizontal Gabor contrast discrimination task at the transfer location led to a large degree of transfer from the primary task to this transfer location. The benefits from cross training also depended on performance levels during training, with less benefit from training at threshold performance levels.³⁴

Understanding the factors that influence the balance between specificity and transfer is important not only for the theoretical understanding of visual perceptual learning, but also for its practical applications in visual rehabilitation, where transfer of learning to untrained tasks or new context is critical. Relative to the many potential factors that could influence the extent of specificity and transfer, the scope of existing investigations has been relatively limited. Additional systematic empirical and theoretical investigations on many additional factors in a broad range of perceptual learning tasks are necessary to identify causal factors of the extent of specificity and transfer.

Mechanisms of perceptual learning

Human perception is limited by various sources of noise such as intrinsic stimulus variability (such as quantum fluctuations), receptor sampling errors, randomness of neural responses, and loss of information during neural transmission. From this perspective, any perceptual task involves discerning relevant signal from noise and any performance improvement such as perceptual learning reflects an improved signal-to-noise ratio in perceptual processing.³ Here we discuss sources of noise, the mechanisms and neural plasticity underlying improved signal-to-noise ratio, and two broad theories of how visual perceptual learning improves performance.

Modeling internal and external noise

Regardless of the perceptual task, perceptual learning reflects improved signal-to-noise ratio in perceptual processing, either by improving the signal or reducing the noise of the observer. Measuring visual perceptual learning using stimuli embedded in systematically varying amounts of external noise and modeling the observer as a noisy perceptual decision maker with both additive and multiplicative internal noises¹¹⁶ reveals the mechanisms of visual perceptual learning.^9,40 In the perceptual template observer model, there are three sources of noise: the external noise added to the signal stimulus (the Gabor) by the experimenter, the additive internal noise of the observer which does not vary with the input stimulus, and the multiplicative internal noise of the observer which increases with the amount of energy in the input stimulus. The signal-to-noise ratio is determined by the ratio of the signal contrast (or differences) and the pooled standard deviation of the external, additive and multiplicative internal noises. Learning-induced stimulus enhancement in the perceptual system amplifies the input stimulus, including both the signal and external noise as well as the multiplicative internal noise, relative to the additive internal noise (Fig. 4a). It is equivalent to additive internal noise reduction and can only improve performance when the external noise is low (Fig. 4c). Learning-induced external noise exclusion filters out external noise (Fig. 4b) and improves performance when the external noise is high (Fig. 4d). Learning-induced multiplicative internal noise reduction reduces the multiplicative noise without changing signal contrast, external noise, and additive internal noise (Figs. 4ab). It improves performance in the full range of external noise conditions (Fig. 4e).

Figure 4.

Three ways to improve signal-to-noise ratio. (a) A signal Gabor in additive and multiplicative internal noises, without external noise. Stimulus enhancement increases the contrast of the Gabor and multiplicative noise relative to the (unchanged) additive internal noise. Multiplicative internal noise reduction reduces the multiplicative noise without changing signal contrast and additive internal noise. (b) A signal Gabor embedded in high external noise and additive and multiplicative internal noises. External noise exclusion filters out the external noise without changing signal contrast. Multiplicative internal noise reduction reduces the multiplicative internal noise without changing signal contrast, external noise, and additive internal noise. Signature threshold contrast vs external noise contrast functions for stimulus enhancement (c), external noise exclusion (d), and multiplicative internal noise reduction (e). The dotted and solid curves represent pre-training and post-training functions, respectively.

Experiments using the external noise paradigm have documented a mixture of stimulus enhancement and external noise exclusion associated with visual perceptual learning in orientation identification, motion direction discrimination, face identification, and hyperacuity tasks,^{9,40,86,96,117} stimulus enhancement in second-order letter identification,¹¹⁸ and external noise exclusion in orientation discrimination in the fovea.³² Training with stimuli in zero external noise engaged both stimulus enhancement and external noise exclusion, whereas training in high external noise only engaged external noise exclusion.^39,119

Two mechanisms, stimulus enhancement and external noise exclusion, either in conjunction or separately, have explained all the data in external noise studies of perceptual learning and visual rehabilitation across a wide range of tasks. These mechanisms are fully compatible with the reweighting theory of perceptual learning and might help neuroscientists search for plasticity in the brain.

Theories of perceptual learning

Two theories, representation enhancement and information reweighting, have been proposed to account for performance improvements in visual perceptual learning.^{4,9,18,82,120–126} Although they both improve signal-to-noise ratio in perceptual learning, representation enhancement does so by changing the responses or tuning characteristics of neurons in early visual cortical areas, whereas information reweighting does so by up-weighting the relevant and down-weighting the irrelevant representations from early visual cortical areas during perceptual decision without changing the representations themselves. In its strong form, the representation enhancement account assumes that perceptual learning enhances stimulus representations in the primary visual cortex.^4,125 By contrast, the reweighting account assumes that perceptual learning improves the readout of task-relevant information that could occur early or late in the visual system, changes the lateral interactions within a layer, and/or introduces feedback from higher levels back to lower levels.¹²⁷ Although reweighting of information from one area to the next could resemble representation change in the multi-area hierarchical visual system, there is a fundamental functional difference between the two theoretical positions. Representation enhancement is persistent and could influence tasks that are unrelated to the trained task but rely on the same representations, whereas selective reweighting is restricted to the trained task or similar tasks.^3,98,99

Both representation enhancement and reweighting could contribute to visual perceptual learning,^{18,83,128,129} but selective reweighting seems to be the dominant component.³ The strongest behavioral evidence for representation enhancement is the specificity of learned behavioral improvements to stimulus features coded in early visual cortical areas, which can also be accounted for by selective reweighting. In addition, even if representation enhancement were to occur in the earliest visual representations, reweighting of the evidence from the altered representation responses is still required to optimize performance. In this sense, the two theories are not wholly exclusive of one another.^120,130

A comprehensive review of neurophysiological perceptual learning studies concluded that the bulk of existing evidence supports relative stability of visual representations in early visual cortical areas.³ In studies reporting that learning induced neuronal representation changes, the observed neural plasticity in early visual cortical areas during passive viewing or in control tasks—which reflects persistent representation enhancement—only accounted for a small fraction of behavioral improvements.^131,132 Evidence of neural plasticity was strongest in higher visual areas under conditions of active task performance, reflecting transient task-specific top-down influence, not persistent representation enhancement.^133–135

In brain imaging studies, neurofeedback [G] designed to specifically target functional activation patterns in V1 (early visual cortex) in the absence of visual stimuli improved behavioral performance and decoding from the neural signal in V1 but not in V4 (a higher visual area). The results showed that V1 can be changed by neurofeedback.¹³⁶ By contrast, one perceptual learning study found that training increased decoding accuracy of functional resonance imaging activity patterns in higher but not early visual cortical areas,¹³⁷ and another study found no change in the overall level of the BOLD responses in V1-V4_.¹³⁸ These results suggest that, although neurofeedback could improve stimulus representation in early cortical areas, perceptual learning is consistent with plasticity in higher-level visual cortical areas by reweighting information from early visual cortical areas without altering their representations.

Representation enhancement has so far remained a verbal theory; virtually all existing computational models of visual perceptual learning are based on selective reweighting that improves the readout of sensory information in decision (Box 1).^{8,9,26,80,121,127,139–142} One modeling study showed that trial-by-trial learning through reweighting can account for all the observed mechanisms of perceptual learning in representative experimental data manipulating external noise.¹⁴³

Specificity, the primary behavioral evidence for theories of representation enhancement, can be accounted for by the reweighting theory. Physiological and brain imaging studies have also supported the conclusion that visual perceptual learning largely reflects reweighting of sensory representations of visual stimuli, with early sensory representation enhancement accounting for a small fraction of behavioral performance improvements.^{3,131,133,144} Although the pattern of learning-related changes in neural data support the idea that reweighting is a dominant form of plasticity, new physiological evidence is required to trace the reweighted connections between brain areas.

Neural plasticity

Retinal location specificity of visual perceptual learning originally suggested the involvement of early visual cortical representations because of their small receptive fields.⁷¹ These data led to strong claims of plasticity in early visual cortical areas underlying these improvements; a change in visual performance would seem to imply plasticity in the region representing the visual stimulus. However, an analysis of training and transfer tasks with a simple neural network suggests that the involvement of representations encoded in a cortical area does not imply that plasticity happens in that area.²⁶

Although it cannot distinguish representation enhancement and reweighting theories of visual perceptual learning, specificity is a powerful behavioral indicator of the cortical level(s) involved in learning. Specificity to retinal location, eye, or stimulus features indicates involvement of representations at a level that preserves those properties. By contrast, transfer or generalization over location, eye, or stimulus features indicates involvement of representations at a level that abandons those properties.

Essentially all the observed specificity in the literature is consistent with plasticity of the connections between sensory representations and perceptual decision, and does not require plasticity at the site of the sensory representations (such as in early visual cortex).^3,26 In specificity to retinal location, eye, and stimulus, the stimuli in the training and transfer tasks are represented by different neurons in early visual cortical areas, with independent connections to perceptual decision. Learning in the training task by changing either the relevant neural representation or the connections to decision would not have any impact on the representation or connections of the transfer task and therefore result in specificity. Thus, specificity cannot indicate whether visual perceptual learning involves plasticity in early visual areas or improved connections from early visual areas to areas involved in decision-making. In task specificity, the stimuli in training and transfer tasks are represented by the same sensory neurons connected to different perceptual decisions. Specificity implies plasticity of the connections between sensory and decision areas rather than within sensory neurons.^26,120 By contrast, any transfer suggests that higher-level visual representations invariant to location, eye, or stimulus features are involved in perceptual learning. In these higher-level visual representations, stimuli from different locations, eyes, or with different features are represented by the same neurons. Improved representations or their connections to perceptual decision from training with a stimulus in one location, one eye, or with one stimulus feature would automatically benefit perceptual performance with stimuli in other locations, eye, or with other stimulus features.

Learned plasticity must be balanced by stability to optimize behavior over many tasks and environmental contexts.^40,145 For instance, retuning representations in the early visual cortex to enhance performance in one task would impact performance in many other tasks that use the same representations. By contrast, plasticity based on selective reweighting of task relevant connections in a multiplexing cortical organization,¹⁴⁶ in which different tasks involve independent connections from sensory areas to decision, could help maintain stability during visual perceptual learning of multiple tasks over time. Deep convolutional neural networks might provide a promising way to investigate the tradeoff between plasticity and stability (Box 2).

Box 2. Deep convolutional neural networks.

Deep convolutional neural networks are developed to perform perceptual functions such as image recognition. With large corpus of training images and associated labels, a deep convolutional neural network learns to adjust connection weights between different layers and can perform image recognition with very high accuracy. Although deep convolutional neural networks are developed to solve classification problems in artificial intelligence rather than to model human behavior, they face many of the same theoretical issues encountered in visual perceptual learning, such as specificity versus transfer and the trade-off between plasticity and stability.

Deep convolutional neural networks have also been used to model perceptual learning.^191,192 There are a few major differences between deep convolutional neural networks and the integrated reweighting theory (Box 1). First, whereas the representations and connection weights in deep convolutional neural networks are learned through training from scratch, the initial representations and connection weights in the integrated reweighting theory are based on known properties of the visual system, reflecting the fact that human observers can perform most visual perceptual learning tasks above chance before training. In fact, early layers of the deep convolutional neural networks after extensive training represent visual inputs in similar ways as early visual cortical areas.¹⁹³ Second, whereas deep convolutional neural networks are trained with labeled images in a supervised mode, the integrated reweighting theory uses an augmented Hebbian rule and can learn without supervision, corresponding to the fact that learning often occurs in visual perceptual learning even in the absence of external feedback. The integrated reweighting theory is also more robust to small perturbations of the visual stimuli than the deep convolutional networks. Third, whereas only the highest representation layer is connected to decision-making in deep convolutional neural networks, every layer in the integrated reweighting theory is connected to decision-making. The types and degrees of invariance in different layers of the integrated reweighting theory account for the extent of specificity and transfer over a wide range of visual perceptual learning studies. On the other hand, deep convolutional neural networks often ‘yoke’ weight changes in different locations in each layer, assuming that training in one location propagates to analogous parts of the network. Such an architecture leads to complete location transfer. Furthermore, deep convolutional neural networks have not incorporated meaningful treatments of internal noise, a fundamental property of the human visual system. Despite these differences, cross fertilization of research on perceptual learning and deep convolutional neural networks might help advance both fields.

Applications to rehabilitation

Visual perceptual learning studies in the laboratory have demonstrated brain plasticity and large improvements in perceptual performance, inspiring a quest to translate the procedures of visual perceptual learning to rehabilitation therapies. Many clinical populations could potentially benefit from perceptual learning, including patients with amblyopia, myopia, presbyopia, optical aberrations, low vision, cataract, and cortical blindness.^{24,25,147–150} In all these cases, perceptual learning can be used to reduce visual deficits by training the brain to improve the signal-to-noise ratio in perceptual processing, without the need to alter physical limitations of the eye (such as the elongated axial length of the eye in myopia).

Lessons from clinical applications have also greatly enhanced the scientific understanding of visual perceptual learning. Just as theories and procedures from visual perceptual learning studies of normal adults can provide a launch pad for the development of rehabilitation therapies, translation work faces many new challenges not encountered in laboratory studies and can lead to different theoretical accounts of patient behavior as well as new training procedures.^3,27 We focus on amblyopia, myopia, and low vision, where visual perceptual learning has exhibited some promising outcomes.

Amblyopia

Amblyopia is the most common cause of visual impairment in childhood (present in 2–5% of the population) and is characterized by poor spatial vision (the ability to perceive spatial patterns of light) in one (‘unilateral amblyopia’) or both eyes (‘bilateral amblyopia’) without detectable structural or pathological abnormalities.^25,151 Conventionally, children with unilateral or asymmetric bilateral amblyopia are treated by patching or unilateral atropine application to block or blur the vision of the eye with better vision to decrease its use and promote use of the eye with worse vision. Adults with amblyopia are not treated in the clinic because the conventional wisdom is that the critical period for spatial vision development ends at age 8, after which patching and atropine treatments are no longer effective.¹⁵²

Visual perceptual learning has been evaluated as a treatment mostly for adults with amblyopia.²⁵ Training the amblyopic eye using monocular tasks, such as Gabor contrast detection, Gabor orientation discrimination and hyperacuity, significantly improved performance of the amblyopic eye in the trained task without damaging performance of the fellow eye in adults with amblyopia.^{5,25,111,153–157} In particular, training the amblyopic eye in Gabor contrast detection improved the signal-to-noise ratio via stimulus enhancement and external-noise exclusion,¹⁵⁸ transferred to a broader range of spatial and temporal frequencies than the equivalent training in observers with normal vision,^{10 159} and had a long retention period.⁵ Using simple visual tasks for adults with amblyopia and video games for both children and adults with amblyopia, training with simultaneous presentation of contrast-enhanced images in the eye with worse vision and/or contrast-reduced images in the eye with better vision improved performance in the trained tasks as well as stereo acuity and visual acuity.^{157,160–165}

Reviews and meta-analyses of laboratory studies concluded that 6–30 hours of visual perceptual learning in a wide range of tasks transferred to 0.1 to 0.2 log minimum angle of resolution [G] (logMAR) visual acuity improvements in amblyopia, regardless of the task, whether training was monocular or binocular, age (for adults) or the type of amblyopia.^{25,153,154,157,166} In comparison, 120 hours of wear an eye patch is required to achieve a 0.1 logMAR acuity improvement. However, randomized clinical trials on binocular training generated visual acuity improvements comparable to patching.^167,168 The many potential contributing factors of the different results from the small-scale laboratory studies and large-scale randomized clinical trials, including inclusion criteria, placebo effects, experimental protocol, and patient compliance, are still under active investigation.²⁵

Myopia

The prevalence rate of myopia is 43% in the US population below age 50 and 80%–90% in young adults in East Asia.^169,170 Visual perceptual learning has been used to improve cortical processing of blurry myopic images on the retina to improve myopic vision. Training mildly to moderately myopic individuals in Gabor detection significantly improved contrast-sensitivity and produced about 0.16–0.25 logMAR improvements in acuity, with no change in refractive error [G].^171–174 By measuring the threshold contrast versus external noise contrast functions (Fig. 4) before and after training, a study found that these improvements reflected increased signal to noise ratio via a combination of stimulus enhancement and external noise reduction.¹⁷⁴ Although visual perceptual learning has shown some promising results in adults with mild to moderate myopia in several laboratory studies, large-scale clinical trials are necessary to fully evaluate its efficacy adults with mild to high myopia and children with myopia. Future studies should also evaluate whether visual perceptual learning can slow the progression of myopia in children.

Low vision

Low vision refers to vision loss that cannot be corrected with glasses, contact lenses or surgery, typically caused by age-related macular degeneration, diabetic retinopathy, or glaucoma. People with low vision might have blind spots, poor night vision, and/or blurry sight but are not fully blind.¹⁷⁵ One study showed that training in a visual search task led to improved performance for patients with severe to profound low vision.¹⁷⁶ Low vision patients with central vision loss often develop one or more preferred areas in the periphery as their ‘new fovea’, which is repeatedly aligned with a visual target for a specified task and serves as the oculomotor reference. Several studies showed that visual perceptual learning on orientation discrimination, contrast detection and letter recognition improved visual acuity and contrast sensitivity and reduced visual crowding in the preferred retinal locus of patients with low vision.^177–180 Furthermore, visual perceptual learning promoted the selection and stabilization of a good preferred retinal locus (that can precisely and stably align with visual targets) in normal observers with simulated central vision.^181–183 One review suggests that a more holistic intervention approach, taking into account eye movement planning, cognitive control mechanisms and visual perceptual learning, is necessary to improve rehabilitation in low vision.²⁴ Whereas contrast sensitivity and visual acuity at the preferred retinal locus may be improved with visual perceptual learning, oculomotor control may be improved with eye tracking technology or computerized programs, and cognitive control can be trained with visual search and/or tasks engaging various components of attention.

Applications of visual perceptual learning in clinical populations have revealed its potential for partial remediation of specific visual functions. However, many challenges remain.^25,27 First, specificity can limit the benefits of training to trained stimuli and tasks or those highly similar to them. Second, many existing studies have used relatively simple laboratory training tasks. Translational research on perceptual learning could benefit from the inclusion of more real-world training tasks and a test battery to assess the generalizability of proposed training protocols. Such a battery should include a broader range of visual tests, from basic visual features to daily visual functions, such as reading, driving, or mobile device use. Full testing of medically relevant training interventions might require large sample randomized clinical trials and approval by regulatory agencies to truly establish the efficacy of visual perceptual learning-based treatments.^25,27,157

Summary and future directions

Perception can be powerfully improved by training or practice, even in adults. Visual perceptual learning occurs at multiple levels of a complex hierarchical brain network by improving the signal-to-noise ratio in perceptual processing either through enhancing the stimulus or excluding external noise. One of the distinctive characteristics of visual perceptual learning is that the benefits are often specific to the trained stimulus and task, with the magnitude of learning and extent of specificity mediated by multiple task and stimulus factors. Computational models that reweight information from multiple visual areas to decision in a hierarchical brain network have accounted for a wide range of visual perceptual learning phenomena. In addition, visual perceptual learning has led to performance improvements in patients with visual disease and can offer effective rehabilitation in some visual conditions. Despite the tremendous progress made in our understanding of visual perceptual learning over the last three decades, there are still many unanswered questions.

Improved measurement and modeling of the trial-by-trial learning curve has the potential to expand understanding of perceptual learning mechanisms.^{184,185 186} Incorporating regularities across observers and conditions, hierarchical Bayesian models provide more accurate and precise estimates of the parameters that define the learning curve. The fine-grained assessment captures rapid learning, within-session deterioration, and abrupt between-session performance changes.³⁷ These methods can also provide more accurate and precise assessments of specificity and transfer because of better assessment of the initial performance in training and transfer tasks than block or session measures. These new methods can be applied in a wide range of studies to improve the use of data from individual participants to better understand learning in populations. A more efficient trial-by-trial assessment of the learning curve would also permit the estimation of learning curves with fewer assessment trials. Future work could use a mixture of supra-threshold trials for better training and near-threshold trials to assess performance improvements, a more efficient learning procedure.

Many factors impact the magnitude of performance improvements and extent of transfer in visual perceptual learning. It is impossible to find the optimal combination of the factors by evaluating all possible combinations through experimentation or even computational simulations. To obtain the optimal combination and therefore the optimal training protocol, an optimization framework could be used,¹⁸⁷ which consists of a well-specified objective function such as maximum learning or transfer, a model of trial-by-trial performance in visual perceptual learning and transfer, and a powerful engine to efficiently search for the optimal combination. In addition, experiments are necessary to verify the key assumptions and evaluate the performance of the components of the optimization framework, especially the perceptual learning model.

Another avenue for future research is more realistic brain network models. Although existing models have successfully accounted for a wide range of observed behavioral phenomena in visual perceptual learning, they are necessarily abstract and relatively simplified. The next generation of visual perceptual learning models could directly incorporate the architecture of the visual system, from the optics of the eye to the retina, lateral geniculate nucleus, and visual cortical areas, as well as attention, reward, and decision circuitry. They might also incorporate recurrent, feedforward and feedback connections within and between brain areas as well as neuronal properties in each area, especially stochastic properties of neural processing. Such developments will require coordinated investigations across computational neuroscience, system neuroscience and sensory psychology.

Most laboratory perceptual learning studies have focused on adults, with few systematic studies of perceptual learning in children, whose brains are still undergoing development and are more labile. Animal studies show that the visual system is much more plastic during the critical period, which can be re-opened through dark rearing [G].^188,189 There is tremendous interest in understanding the interaction between perceptual learning and development, especially the possibility of applying perceptual learning during a re-opened critical period to take advantage of the enhanced plasticity to improve sight in patients with eye disease. However, children might be more susceptible to unforeseen side effects during development and many ethical issues must be addressed in this work.

Visual perceptual learning, reflecting plasticity of the visual system, is a fundamental property of the human brain. Tremendous progress has been made in understanding the phenomenology, mechanisms, and applications of visual perceptual learning over the last three decades. This research has in turn greatly enriched the understanding of the visual system. A new generation of measurement technologies, data analytical methods, applications of the optimization framework, and modeling efforts, will further deepen the understanding of visual perceptual learning and extend its application to child development and rehabilitation.

Glossary

Fovea:

The location in the very center of the retina where eyesight is sharpest

Hyperacuity:

A sensory capability beyond classical visual acuity limits, such as the ability to detect a misalignment of lines that is considerably smaller than the diameter of a foveal cone receptor

Neurofeedback:

Feedback based on measurement of real-time brain activities, used to teach self-control of brain functions to participants

Log minimum angle of resolution:

The logarithm of the angle at which two objects such as two points subtended at the eye are just perceived as separate

Refractive error:

A type of vision problem, including nearsightedness, farsightedness, astigmatism and presbyopia, caused by the abnormal shape of the eye that keeps light from focusing correctly on the retina

Dark rearing:

A form of rearing with visual deprivation in which the animal spends more time in total darkness than the normal day-night cycle