Visual Perceptual Learning and Models

Abstract

Visual perceptual learning through practice or training can significantly improve performance on visual tasks. Originally seen as a manifestation of plasticity in the primary visual cortex, perceptual learning is more readily understood as improvements in the function of brain networks that integrate processes including sensory representations, decision, attention, and reward and balance plasticity with system stability. This review considers the primary phenomena of perceptual learning, and theories of perceptual learning and its effect on signal and noise in visual processing and decision. Models, especially computational models, play a key role in behavioral and physiological evaluation of the mechanisms of perceptual learning, and for understanding, predicting, and optimizing human perceptual processes, learning, and performance. Performance improvements resulting from reweighting or readout of sensory inputs to decision provide a strong theoretical framework for interpreting perceptual learning and transfer that may prove useful in optimizing learning in real world applications.

INTRODUCTION

Visual perceptual learning is the improvement in visual task performance with practice or training (Sagi 2011). It reflects learning and plasticity in the visual system and a network of other brain substrates of behavior. Substantial improvements in visual task performance can occur in adults whose cortical organization and function are developmentally mature, where the architecture of the visual system—absent major injury and reorganization—is relatively stable (Wandell & Smirnakis 2009). Although research has heavily focused on the development of perceptual expertise in adults (Lu et al. 2011), visual perceptual learning contributes to functional improvements during development (Atkinson et al. 1977, Gibson 1969), can improve visual performance during aging (DeLoss et al. 2015), and plays an important function in visual rehabilitation (Lu et al. 2016). Finally, the state of the perceptual system depends on experience, and cannot be fully understood without understanding its plasticity.

Human perceptual processes are a necessary gateway to experience and integral to planning and executing behavior. The visual system is a complex processing engine, with many areas and modules that coordinate a complex flow of perceptual information (Van Essen et al. 1992). The visual system, like many sensory systems, seems to have evolved to support processing of the important stimulus cues in the environment (Geisler 2008). However, even after millions of years of evolution, it continues to improve through development and through experience. Human visual functions improve considerably during a developmental period starting in infancy. For normal adults, performance in many visual tasks may be far from optimal and can be improved with the right kind of training or practice (Lu et al. 2011).

Observations of perceptual learning date back to early in the study of perception. William James describes improvements in performance with practice in his chapter on discrimination and comparison, citing work by Volkmann and Fechner on improvements in two-point discrimination on skin from the mid-1800s (James 1890). Stratton studied the role of experience with prism distortions of visual input in the late 1800s (Stratton 1897). Perceptual learning and its role in visual development was studied by E. Gibson starting in the 1950s (Gibson 1969). There were many observations of expertise in naturalistic tasks such as wool sorting, wine tasting etc. More controlled laboratory studies of perceptual learning refocused in the late 1980s, and many features of perceptual learning, its specificity to stimuli and tasks, and its mechanisms have been discovered during the last nearly three decades.

OBSERVING PERCEPTUAL LEARNING

Learning in Perceptual Tasks

Perceptual learning has been documented in virtually all tasks and sensory modalities—although performance in a few tasks is relatively unchanged by practice, perhaps because they are so common in everyday life. A number of reviews have considered this now-extensive body of work, including (Adini et al. 2004, Ahissar & Hochstein 2004, Fahle & Poggio 2002, Fine & Jacobs 2002, Lu et al. 2011, Vogels 2010, Watanabe & Sasaki 2015).

Perceptual learning occurs for different kinds of tasks at different levels of visual analysis (figure 1). It improves detection or discrimination for single features such as orientation (Dosher & Lu 1998, Dosher & Lu 1999, Schoups et al. 1995, Vogels & Orban 1985), spatial frequency (Bennett & Westheimer 1991, Fiorentini & Berardi 1981), phase (Dosher et al. 2010, Fiorentini & Berardi 1980), contrast (Adini et al. 2004, Dorais & Sagi 1997, Sowden et al. 2002), color (Casey & Sowden 2012, Özgen & Davies 2002, Thurston & Dobkins 2007), acuity (Bennett & Westheimer 1991, Westheimer 2001), and hyper-acuity (Crist et al. 1997, McKee & Westheimer 1978, Poggio et al. 1992). It improves pattern discrimination in tasks involving compound stimuli (Fiorentini & Berardi 1980, Fiorentini & Berardi 1981), textures (Ahissar & Hochstein 1993, Karni & Sagi 1991), depth (Fendick & Westheimer 1983, Ramachandran & Braddick 1973), and motion (Ball & Sekuler 1982, Lu et al. 2006, Lu et al. 2005, Matthews & Welch 1997, Watanabe et al. 2002). And it improves identification of objects and natural scenes such as faces and entities (Gauthier et al. 1998, Gold et al. 1999), shapes and objects (Kourtzi et al. 2005, Nazir & O’Regan 1990), and biological motion (Grossman et al. 2004, Jastorff et al. 2006). In many of these cases, the literature is extensive; we have often provided earlier references.

Figure 1.

Visual perceptual learning improves task performance measured in different ways: improving percent correct discrimination (a), contrast thresholds (b), or feature difference threshold differences (c) (hypothetical data and generating exponential learning curves). (d) Difference thresholds (arcsec) from a line offset hyperacuity task for vertical, horizontal, and oblique layouts (data from McKee & Westheimer, 1981), fitted exponential learning curves added. Reproduced with permission from B. Dosher and Z.L. Lu.

Perceptual learning can powerfully affect performance, improving accuracy from near chance to more than 90% correct in many two-choice tasks (Ball & Sekuler 1982, Fiorentini & Berardi 1980, Poggio et al. 1992). For this reason, any characterization of visual functions based on testing will depend upon the level of expertise. Finally, training effects are sufficiently large that they can contribute to performance in practical domains. On the other hand, there are cases where perceptual learning does not alter performance (Sagi 2011); failure to improve often involves already practiced judgments such as training at fovea with prototypical features in easy viewing conditions (Lu & Dosher 2004), or when potentially conflicting judgments are intermixed or roved (Herzog et al. 2012, Yu et al. 2004).

Visual perceptual learning has been used to improve visual performance in visual conditions, including amblyopia (Levi & Li 2009, Li et al. 2013, Xi et al. 2014), myopia (Durrie & McMinn 2007, Yan et al. 2015), aging (DeLoss et al. 2015), presbyopia (Polat et al. 2012), low vision (Liu et al. 2007, Yu et al. 2010), cortical blindness (Huxlin et al. 2009, Kasten & Sabel 1995, Nelles et al. 2001), and rehabilitation after surgical interventions (Huber et al. 2015, Kalia et al. 2014). It has also been applied in education (Kellman & Massey 2013, Merzenich et al. 1996, Strong et al. 2011) and training of visual expertise (Deveau et al. 2014, Gauthier et al. 1998, Sowden et al. 2002).

Perceptual learning often involves thousands of trials of practice over days or weeks (Dosher & Lu 1998), although sometimes a few exposures of easy stimuli accelerate learning (Liu et al. 2012, Rubin et al. 1997), and in some domains initial learning occurs within a few dozens trials (Ramachandran & Braddick 1973). Some studies find that REM sleep was critical for perceptual learning in texture discrimination (Karni & Sagi 1991, Mednick et al. 2003). Training effects can persist for periods up to years (Karni & Sagi 1993). In sum, perceptual learning is a major phenomenon of adult plasticity with important theoretical and practical implications.

Specificity and Transfer

The specificity of visual perceptual learning (Karni & Sagi 1991) is one of its hallmark characteristics (figure 2). Specificity—in which learned improvements are lost when the stimuli or task is altered—has been reported for orientation, spatial frequency, motion direction, pattern, and even (significantly) location in the visual field (Ball & Sekuler 1982, Dosher & Lu 1999, Fahle & Edelman 1993, Fiorentini & Berardi 1980, Karni & Sagi 1991, Schoups et al. 1995). For example, training to detect a small patch of differently oriented lines in the lower right quadrant does not fully transfer to texture processing in other quadrants (Karni & Sagi 1991). The specificity of trained improvements to a portion of the visual field were considered especially salient, leading some to infer that perceptual learning reflects plasticity in early visual cortex, V1, which has small retinotopic receptive fields.

Figure 2.

Learning in a training task can express varying benefits for a transfer task. (a) Hypothetical learning curves showing full specificity, partial specificity, and full transfer; (b) corresponding patterns in bar graphs that often summarize these results; and (c) data from an experiment training texture detection in different quadrants of the visual field that shows significant specificity to retinal location and partial transfer (data from Karni & Sagi, 1999, figure 1). (a)-(c) reproduced with permission from B. Dosher and Z.L. Lu. (d) with permission from XXXX

While the literature emphasized the specificity of visual perceptual learning, often it is a graded phenomenon with some specificity and some transfer of trained improvements to other stimuli, tasks, or locations (Dosher & Lu 2009). Generalization is more important in practical applications, such as the development of expertise or remediation, where benefits beyond the training conditions are valuable. We now know that the relative specificity versus transfer of training can depend on several factors, including the processing level of the trained task (Fine & Jacobs 2002), the task difficulty (Ahissar & Hochstein 1993, Liu 1999), precision of the transfer task (Jeter et al. 2009), the extent of training (Jeter et al. 2010), the state of adaptation induced by training (Censor et al. 2006), and the exact training and transfer procedure (Hung & Seitz 2014, Xiao et al. 2008). More demanding tasks tend to experience more specificity.

Feedback and Reward

Feedback occurs when the learning environment provides information about the quality or value of behavior. The form of feedback can differ, and these differences can affect the magnitude and speed of perceptual learning. Training may use full feedback about the desired response on each trial, occur on a subset of trial, be misleading, or feedback may be unavailable (Dosher & Lu 2009). Almost all experimental investigations of perceptual learning involve two-alternative choices (e.g., left/right, same different) and trial-by-trial accuracy feedback. However, learning can occur with only block feedback (Herzog & Fahle 1997, Shibata et al. 2009) or without any feedback if performance before training is high enough (Fahle & Edelman 1993, Herzog & Fahle 1997, Liu et al. 2010, McKee & Westheimer 1978, Petrov et al. 2006). If performance accuracy before training is low, observers may not learn without feedback (Liu et al. 2010, Rubin et al. 1997), or the rate of learning increases with feedback (Crist et al. 1997). Reverse or random feedback can prevent learning (Aberg & Herzog 2012, Herzog & Fahle 1997, Herzog & Fahle 1999), while exaggerated (positive) block feedback can change its rate (Shibata et al. 2009). In some unusual demonstrations, feedback in the absence of a stimulus can alter performance (Choi & Watanabe 2012, Shibata et al. 2012). In short, learning can occur in the absence of feedback in certain situations, yet feedback is important when the task is difficult and initial performance is poor. Trial-by-trial feedback is more effective than block feedback, and inaccurate feedback can disrupt learning.

Physical rewards can result in perceptual learning in the absence of verbal instructions (Seitz et al. 2009). Yet explicit rewards, and especially the systematic effects of the magnitude of rewards on perceptual learning, are just beginning to be investigated (Zhang et al. 2016). A model with the right learning rule has the potential to systematize and predict these varied effects of feedback and reward.

Selection by Task and Attention

Perceptual learning balances stability with plasticity in part through selectivity. Real world sensory stimulation is rich, containing many potential cues for guiding behavior. Yet generally only task relevant stimuli, features, or locations participate in learning (Ahissar & Hochstein 1993, Fahle & Morgan 1996, Shiu & Pashler 1992). However, task-irrelevant learning sometimes occurs for extraneous stimuli appearing in temporal proximity to a training stimulus if they are subliminal (Gutnisky et al. 2009, Watanabe et al. 2002).

In addition to task-relevance, attention may select what is learned. Although there are claims that attention to a stimulus is required for learning (Ahissar & Hochstein 2004, Dolan et al. 1997, Gilbert et al. 2001), only a few studies have explicitly manipulated attention and evaluated learning in attended and unattended conditions (Mukai et al. 2011, Xu et al. 2012). Conversely, the functional importance of attention in determining task performance can be reduced through extensive perceptual training (Dosher et al. 2010). Selection by task or by attention selectively reweights only those stimulus representations or attended features that are relevant to the trained task.

MODELS OF PERCEPTUAL LEARNING

Even the simplest task of detecting or discriminating a perceptual stimulus involves a network of processes or brain regions supporting sensory processing, decision, action selection, top-down task relevance, attention, and processing of rewards or feedback. Each of these sensory and cognitive processes may be engaged during natural behavioral episodes and in each experimental trial. Learning alters processing, generally to improve performance. Although early explanations focused on plasticity in the sensory cortices, perceptual learning must engage multiple processes, levels, and brain areas (Kourtzi 2010, Kourtzi et al. 2005).

Maintaining stability in the face of plasticity, the plasticity-stability dilemma, constrains how the system learns perceptual tasks (Dosher & Lu 2009). Plasticity of visual system is normally considered an advantage associated with performance improvements, yet must be balanced with maintaining stability in standard visual functions. Too much plasticity could result in catastrophic forgetting of one task or set of stimuli by training on another (French 1999, Grossberg 1987), and an inability to optimize several tasks simultaneously.

Several conceptual frameworks for perceptual learning have been proposed, including the primary visual cortical plasticity theory (Karni & Sagi 1991), the reverse hierarchy theory (Ahissar & Hochstein 1993, Ahissar & Hochstein 2004), the reweighting model of perceptual learning (Dosher & Lu 1998), and the dual plasticity model (Watanabe & Sasaki 2015). Representation enhancement theories of perceptual learning identify changes in early visual areas such as V1 as the substrate. Selective reweighting theories of perceptual learning promote stability by improving readout from sensory representations that remain largely unchanged (Dosher & Lu 1998, Mollon & Danilova 1996). Although task-dependent reweighting alters the inputs (representations) at later stages of processing, stable early sensory representations (e.g. V1) could contribute to maintaining performance in previously learned tasks. And, if sensory representations are altered, further reweighting would be necessary to optimize readout or decoding of the (new) neurosensory evidence (Dosher & Lu 2009). Recent theoretical overviews cite both forms of plasticity, along with attention and reward (Watanabe & Sasaki 2015).

Models can play a powerful role in testing these theories of learning and plasticity and account for complex patterns in the empirical literature. A complete model includes modules for sensory representation, decision, and learning, and possibly attention, reward, and feedback. Another important component is noise, or variability, in the internal responses of the system. The accuracy of performance depends critically on extracting signal from the internal noise in the system responses. A computational model predicts behavioral performance by taking the stimuli as inputs and specifying the computations carried out in each module. It can be as abstract as a set of simple computations or it can mimic the architecture of brain areas and the behavior of neural populations.

The Learning Rule

Learning rules, core to neural network theories of learning, have also been central to models of perceptual learning. In artificial neural networks, the learning rules change the weights from input units representing the stimuli to decision or response units. Network learning theories distinguish purely supervised, purely unsupervised, and hybrid or semi-supervised learning (Barlow 1989, Jordan & Rumelhart 1992, Reed & Marks 1998). Supervised learning requires a teaching signal that specifies full information about the desired responses, while unsupervised learning does not. In semi-supervised learning, full information may be provided on a subset of cases or about accuracy but not the nature of the error. In hybrid learning, the rule may be modified by feedback, attention, or reward.

Empirical findings on feedback can inform learning rules in models. That perceptual learning can occur in the absence of feedback, yet can sometimes benefit from or require feedback implies a hybrid of supervised and unsupervised learning (Dosher & Lu 2009, Herzog & Fahle 1998). One plausible rule is augmented Hebbian reweighting (Petrov et al. 2005, Petrov et al. 2006), which combined unsupervised Hebbian learning with guidance from feedback and bias. Other models, motivated by physiological concepts of reward and reward prediction error, use reweighting through reinforcement learning, a form of weak supervised learning (Law & Gold 2009).

Computational Models of Perceptual Learning

Essentially all current computational models of perceptual learning are reweighting models. They specify a task domain, including stimuli and desired responses, the network architecture, the decision, and how learning rules change the weights between representations and decision. Computational models have been developed for hyperacuity (Herzog & Fahle 1998, Huang et al. 2012, Poggio et al. 1992, Sotiropoulos et al. 2011, Weiss et al. 1993, Zhaoping et al. 2003), orientation discrimination (Petrov et al. 2005, Petrov et al. 2006, Teich & Qian 2003), tilt (Jacobs 2009), motion direction discrimination (Law & Gold 2009, Vaina et al. 1995), and contrast discrimination (Adini et al. 2004) tasks.

For example, the early hyper basis function (HBF) network model learned a hyperacuity task in which observers judged lines as offset either top-line left or right (Poggio et al. 1992). Its feed forward architecture included a stimulus input layer, a representation layer for localization, and a single-unit output layer (left/right). Modeling studies (Weiss et al. 1993) identified self-supervised learning rules and added internal noise to match predictions to behavioral data. A related model learned global motion direction judgments from input motion vectors for individual dot motions using integrated motion direction templates in an intermediate layer to generate left/right decisions (Vaina et al. 1995). In all these computational models, learning alters the decoding of the activity in a stable stimulus input layer, possibly through an intermediate representation layer, to the output decision layer. They use unsupervised, or self-supervised, learning rules to account for learning without feedback or reward, and may incorporate supervision to account for feedback (Herzog & Fahle 1999). The input representations are highly simplified. And in general they were tested against simple empirical data such as a learning curve or pattern of specificity (Weiss et al. 1993).

The augmented Hebbian reweighting model (AHRM) (Petrov et al. 2005, Petrov et al. 2006) is a full model of perceptual learning. It uses a sensory representation module that mimics the spatial-frequency and orientation tuned responses of early visual cortices like V1, including nonlinearities (Carandini et al. 1997) and internal noise, compatible with observer models of signal and noise in perception (see SIGNAL AND NOISE, below). An output or decision unit weights evidence from activations in the sensory representation to make a decision. Another input corrects for bias in recent responses and a teaching signal augments unsupervised Hebbian learning when trial-by-trial feedback is available (see figure 3). Learning occurs through reweighting, changing the weights on stimulus evidence to make a decision. This yields an improved weight structure for the task after training or practice. Parameters specifying the sensory representations, such as orientation or spatial frequency bandwidths, are set a priori from physiology; nonlinearities are estimated once and then held constant. Internal noise and learning rate parameters are varied to fit model simulated data to the behavioral data. The model takes stimulus images as inputs to a decision, and learns on each trial, reprising experiments exactly.

Figure 3.

A reweighting framework for visual perceptual learning takes images as input, processes them in task-relevant representations, makes decisions based on the weighted sum of the relevant normalized noisy representation activations, and learns by changing the weights through unsupervised and supervised learning algorithms. As in the observer models, performance depends on normalization and internal noise. Reproduced with permission from B. Dosher and Z.L. Lu.

The AHRM accounts for many perceptual learning phenomena, including improvements in contrast threshold in multiple levels of external noise (visual ‘snow’) with training (Lu et al. 2010), more efficient pretraining with low than high external noise conditions (Lu et al. 2010), the importance of feedback in learning tasks with low but not high initial performance (Liu et al. 2010), how to improve learning by including high-accuracy training trails (Liu et al. 2012, Petrov et al. 2005), and how false feedback induces response biases (Liu et al. 2014).

Other similar models use representation modules coded for orientation (Sotiropoulos et al. 2011), location (Huang et al. 2012) or motion (Lu et al. 2010). The original AHRM spatial-frequency/orientation module combined with an adaptive precision pooling decision module was used for perceptual learning in tilt judgments (Jacobs 2009). A model developed for monkey behavior learns by reweighting evidence from a motion representation using a reinforcement-learning rule (Law & Gold 2009).

Modeling Specificity and Transfer

Reweighting models have also been used to explain specificity and transfer in perceptual learning. They predict transfer if the learned weight structure of the training task and the (semi-) optimal weight structure for the transfer task are compatible, and predict specificity otherwise (Dosher & Lu 2009). A few studies have modeled transfer within a single retinal location: The HBF model predicted performance for stimuli differing in line spacing or length (Weiss et al. 1993). A related orientation basis function model simulated specificity and transfer for different hyperacuity tasks (Sotiropoulos et al. 2011). Specificity and transfer to other stimuli at the same location was predicted by the compatibility of weight structures for the AHRM (Petrov et al. 2005).

Specificity to retinal location implies dependence on local retinotopic neural representations and raises the question—how can learning be transferred to a separate set of local neural representations? The integrated reweighting theory (IRT) uses a hierarchical multi-location architecture consisting of location-specific representations and a location-invariant representation (Dosher et al. 2013) to predict transfer over retinal locations. It (figure 4) explains why the same stimuli and task in a new location shows partial transfer, while the same task in the same location with new stimuli shows nearly full specificity. Learned weights for the location-invariant representations are valid when the same stimuli and task occur in other retinal locations. Training in the same location is specific if the stimuli or task are switched because the optimal weight structures compete. This framework also accounts for why different tasks can be sometimes be trained in different locations (Liu et al. 2014) and how related tasks in different locations interact. A modified version of the IRT also predicts transfer of training to new locations in a double-training experiment that practiced two different tasks in two locations (Talluri et al. 2015), a transfer phenomenon attributed by some to higher order inference (Xiao et al. 2008).

Figure 4.

A framework for predicting transfers across and interactions between learning in multiple retinal locations based on a hierarchical architecture of sensory representations including relatively location invariant representations. Perceptual learning in one retinal location trains weights between both location-specific and location-invariant representations and decision. Transfer reflects compatibility of optimized weight structures, while specificity reflects independence or incompatibility of optimized weight structures for the training and transfer tasks. Reproduced with permission from B. Dosher and Z.L. Lu.

Future Model Development

Current computational models are necessarily simplified, yet have the potential to be elaborated in many ways. Different stimuli and tasks may require new stimulus representation modules, new forms of classification or decision, and/or more complex system architectures. Top-down attention could amplify activity in sensory representations; reward may require modified learning rules; and feedback between or reweighting within representation modules may be required to account for new phenomena. The impact of correlated and uncorrelated noise in population networks (Bejjanki et al. 2011) could be modeled. As the physiological substrates for perception and visual perceptual learning in different tasks are discovered, computational models may increasingly mimic these processes. Still, existing reweighting models, despite their spare and simplified form, have provided a strong basis for systematizing and understanding many phenomena in visual perceptual learning and visual expertise

SIGNAL AND NOISE

Performance Improvements and Improved Signal and Noise

The human brain devotes specialized brain regions to the processing of visual stimuli and visual decisions (Ungerleider et al. 1998). Analysis by visual cortex is complex, likened to powerful deep learning network models (Yamins et al. 2014). Visual functions and behavior are not, however, perfect. Accurate performance is limited by the signal to noise ratio in the evidence leading to behavior (Lu & Dosher 2008). Perceptual learning, then, improves the signal to noise ratio in perceptual processing (Dosher & Lu 1998).

Each act of visual detection, discrimination, or identification is limited by the ability to extract the appropriate signal and by intrinsic noise in the system and extrinsic noise in the stimuli. Internal or intrinsic noise is the stochastic variability in the neural responses to a stimulus arising at every stage of processing, while extrinsic noise is variability in the stimuli. The signal to noise ratio determines the accuracy and response time of an identification or choice behavior. In neural encoding, a driving stimulus leads to a noisy pattern of firing across populations of neurons that must be decoded. Limiting noise should be an explicit factor in models, whether they are signal detection models of behavior (Green & Swets 1966), computational models of perceptual learning (Dosher et al. 2013), or descriptive or computational models of neural responses (Goris et al. 2014). When perceptual learning improves the signal to noise ratio, this could occur through improvements in the extraction of the signal or reduction of noise, or both. Behavioral and neural methods can be used to analyze changes in signal and noise, and can help to identify the mechanisms of visual perceptual learning and can guide the development of models of perceptual learning. These mechanisms are considered next.

Mechanisms: Psychophysics

Discovering how perceptual learning changes the signal and noise processing in the perceptual system from behavioral evidence requires a model of the observer and systematic psychophysical experiments. Observer models inspired by properties of the visual system systematize the behavior of observers in different testing circumstances by characterizing signal and noise processing in perception. These models initially characterized the observer as a single-channel linear system with an additive noise source (linear amplifier model) (Pelli 1981); they have since been successively elaborated to account for data and now include multiple noise sources, nonlinearity and gain control (perceptual template model (Lu & Dosher 2008), or incorporate multiple sensory channels each limited by noise (Dosher & Lu 1998, Hou et al. 2014). These observer models of the sensory representations and decision can be used to characterize changes due to perceptual learning, and could be further elaborated to include learning rules and other factors.

External noise paradigms can be used to specify observer models, sometimes augmented with double-pass paradigms (Lu & Dosher 1998, Saarinen & Levi 1995). To estimate nonlinearities and other parameters that predict the observer’s performance in all signal and external noise conditions, detection or discrimination thresholds are measured at three or more performance levels, e.g., 65%, 75% and 85% correct in several conditions with different levels of external noise added to the signal stimulus (Lu & Dosher 1999). Double pass paradigms, repeating the identical stimuli, including the exact target and external noise sample, can add constraints on observer models and parameters, especially in the ratio of internal to external noise (Burgess & Colborne 1988). These experiments are also used to measure how perceptual learning changes signal and noise processes to improve behavioral performance by specifying which components of the observer model have been changed during learning.

The perceptual template model (PTM) has been the most widely used observer model in perceptual learning (figure 5). The model includes a perceptual template (channel) that extracts evidence, nonlinearity in transduction, internal noises, and a decision stage to predict behavior. Three mechanisms can improve performance: improved filtering or external noise exclusion by tuning the template; enhancement of the target stimulus or equivalently reduction in internal additive noise; or change in nonlinearity/reduction in internal multiplicative noise (Dosher & Lu 1999). Each learning mechanisms has a signature pattern of changes in empirical thresholds at different levels of external noise and accuracy (see figure 3). Mechanisms are inferred by comparing threshold versus external noise contrast (TvC) functions early and late in learning and estimating how learning changes the model parameters for that task, i.e., template quality, amount of internal noise. The model and estimated parameters together can then predict the behavior for many different stimuli, and with simple elaborations, for tasks of different precision (Dosher & Lu 1999, Lu & Dosher 2008).

Figure 5.

Perceptual template model (PTM) of the observer and signature changes in contrast threshold versus external noise contrast (TvC) functions for three mechanisms of perceptual learning. The observer model includes (from left to right) a template tuned to the target stimulus, point nonlinearity, multiplicative noise, additive noise, and a decision template. Three mechanisms of perceptual learning correspond to stimulus enhancement (or internal additive noise reduction), external noise exclusion (filtering), and changes in multiplicative internal noise or nonlinearity)—or mixtures of these. Modified from Dosher & Lu (1999), figure 3). Reproduced with permission from B. Dosher and Z.L. Lu.

Perceptual learning has now been studied using TvC curves in several tasks. In most cases, perceptual learning simultaneously improved external noise exclusion—by improving the perceptual template—and reduced internal additive noise (Dosher & Lu 1999, Gold et al. 1999). The magnitudes of the two improvements can be decoupled. Sometimes, pure patterns of improved template (Lu & Dosher 2004) or internal additive noise reduction (Dosher & Lu 2006) occur. Changes in nonlinearity and/or internal multiplicative noise have not been observed. The different mix of mechanisms of perceptual learning revealed by TvC experiments is consistent with reweighting models of learning (see below) (Bejjanki et al. 2011, Dosher & Lu 1998, Lu et al. 2010).

Mechanisms: Physiology

Physiological substrates and mechanisms of perceptual learning are investigated by measuring the amplitude, tuning curves, absolute variability, variability relative to the mean response (fano-factor), and topography of neural response in visual areas (i.e., V1, V4, MT, LIP, etc.) before and after perceptual training—or by fMRI brain imaging. Neural responses can be measured either while actively performing the task, which engages task-induced strategy and attention and top-down modification of neural responses, or under various passive fixation controls or anesthesia (in monkey), which reflect more persistent plasticity of neural responses (Schoups et al. 2001). Population neural response models can estimate the magnitude of behavioral improvement that could be accounted for by changes in neural responses in a given brain area (Ghose et al. 2002, Raiguel et al. 2006, Schoups et al. 2001, Yang & Maunsell 2004).

The substrates of perceptual learning are more complex than the early claims of plasticity in V1 (Karni & Sagi 1991). Under passive recording, subtle changes in V1 tuning curves can occur in neurons tuned near a trained orientation (Schoups et al. 2001), while others (Ghose et al. 2002) find essentially no change in V1 or V2 neurons. Even when changes in the slope of the tuning functions in V1 or V2 occur, they are estimated to account for perhaps only 1/10^th of the observed behavioral threshold reduction. Perceptual learning increased the response and narrowed the tuning in a small subset of V4 neurons tuned near the trained orientations under passive viewing (Raiguel et al. 2006, Yang & Maunsell 2004), but these changes still fell an order of magnitude short of accounting for behavioral improvements. In contrast, changes in V4 neuron responses measured while actively performing the task, decoded by separate optimal Bayes classifiers before and after training, came closer to explaining behavior (Adab & Vogels 2011). And, improved V1 responses during active contour detection following training reflect feedback from V4 and higher areas later in the response interval (Gilbert & Li 2012). Correspondingly, learning altered responses in LIP, but not in earlier MT during a dot-motion direction task (Law & Gold 2008). In other tasks, higher-order areas receiving visual inputs are part of the learning circuits observed during object identification, including IT (Kobatake et al. 1998, Logothetis et al. 1995), lateral PFC (Rainer & Miller 2000), and decision circuitry (Kourtzi 2010). In addition, perceptual learning can change the brain areas used in a task: Training fine depth judgments inoculated coarse depth judgments from MT deactivation by injection (Chowdhury & DeAngelis 2008). Because the brain is an interconnected network, changes observed in one cortical area need not be the only or even the primary locus of learned plasticity (Wandell & Smirnakis 2009).

Finally, training may affect other neural properties, such as reducing the correlations between neurons to yield more independence across neural responses. Such a reduction was found between pairs of dorsal MST neurons in a fine motion task (Gu et al. 2011). One computational study (Bejjanki et al. 2011) confirmed that reweighting neuronal responses from one brain area to the next (e.g., LGN to V1) could account for the TvC improvements observed in Dosher and Lu (1998). Reweighting reduced correlations between neurons at the later area, while only slightly changing the amplitude or tuning of individual neurons. This study suggests that the amplitude and tuning of individual neurons is less informative than the pattern and correlation of neuronal population responses measured in simultaneous recordings. It also found that TvC functions estimated from neuronal responses could be a robust measure of the relevant population properties.

Broader networks engaged during perceptual learning have been evaluated using fMRI in humans (Furmanski et al. 2004, Jehee et al. 2012, Kahnt et al. 2011, Kourtzi 2010, Kuai et al. 2013, Schwartz et al. 2002, Shibata et al. 2012, Zhang & Kourtzi 2010). Training-related changes in primary sensory areas sometimes occur; but there are almost always also changes in many other areas. Multi-voxel pattern classifiers tuned to decode the voxel activity before and after perceptual learning is one method for connecting brain activity to behavior. While a number of design factors and the role of top-down attention and decision complicate interpretation in the current literature, imaging could ultimately identify the brain networks, from sensory areas to decision, attention, and expectation, involved in learning and performing different perceptual tasks.

OPTIMIZATION

Investigations of perceptual learning have contributed to scientific theories of the perceptual system and its plasticity. Another goal is to maximize the magnitude of perceptual learning and transfer through development of optimized training protocols. A training protocol includes selection of the training stimuli, the task(s) used to train, the number and sequence of practice trials, the use of feedback, or reward, etc. Identifying the best training methods is a problem that can be formulated in terms of mathematical optimization (Lu et al. 2016). First, an objective function must specify the goals and grade the outcome; this might include several factors such as the magnitude and efficiency of learning and the desired transfer characteristics, and their relative importance. Next, a generative model predicts the outcome(s) for all protocols within the domain of possible training manipulations. A search algorithm identifies the one or several candidate protocols to maximize the objective function. Optimizing training by intuition and experimentation is prohibitive. A robust computational model can be uniquely valuable in optimization—replacing expensive and time consuming empirical exploration with simulation. Empirical testing may still be required to estimate parameters of the generative model and potentially to improve the model itself. And experimental tests must ultimately validate the best training protocol.

SUMMARY, CHALLENGES, AND FUTURE DIRECTIONS

Since the earliest research in psychology and neuroscience, perceptual learning has been recognized for its important role in perception and performance. Since its resurgence in the late 1980s when it was recognized as a key demonstration of visual plasticity in adults, it has become an important topic of study in the visual sciences. We extract four principles of perceptual learning from this extensive literature:

• Perceptual learning occurs within a complex set of brain networks and may occur at multiple levels.

• Learned plasticity must be balanced by stability in order to optimize the behavior over many tasks and environmental contexts.

• Reweighting evidence from one level to another of representation or within levels is a major and perhaps the major form of perceptual learning.

• Perceptual learning improves the signal to noise ratio limiting human performance either by enhancing signal or reducing noise.

There are many exciting areas for future research. One is to characterize the interaction between perceptual learning and lifespan development. Perceptual learning may have powerful influences on visual function during early childhood (Atkinson et al. 1977), and could aid in maintaining function during normal aging (DeLoss et al. 2015). Both domains are only beginning to be mined, perhaps due to ethical considerations and the potential for unintended consequences, especially in early development.

The development of more sophisticated methods of measuring learning and transfer is another area for future research. The act of measuring visual functions can change the state of the system, analogous to the observer effect in physics. For example, transfer/specificity measures often compare post-training performance to pre-training baselines, yet measuring pre-training baselines causes learning that must be estimated. The magnitude and speed of learned improvements depend upon the mixture of stimuli used while measuring snapshots of performance. Developing better performance measures and understanding the pros and cons of different procedures and how to model them would benefit future applications.

Computational models have already played a powerful role in systematizing a broad range of empirical phenomena and generated new and testable predictions that can guide future basic and translational research in perceptual learning. The convergence between models at the level of learning architectures and learning networks with brain architecture and with cellular physiology could extend our understanding of perception and performance and underlying brain substrates. Multi-modal models that relate computations to behavior on one hand, and measures of brain activity on the other may reveal new principles of learning and plasticity and an improved ability to interpret brain function. Models can also play a key role in optimizing training protocols.

Research in perceptual learning has an increasingly recognized potential for important translation into systems for training expertise or improving remediation and recovery. One of the key challenges is how to improve generalizability of training that otherwise may show undesirable specificity. One recent direction goes beyond simplified laboratory tasks to more complex and realistic tasks and training systems. There have been widespread claims that video games and other digital training apps can enhance learning and generalization, and improve broad visual functions (Green & Bavelier 2015). The roles of reward, pacing, and task variation in these games and apps all deserve investigation that could improve not just translation but reveal new insights in basic science of vision. Finally, as a practical matter, translation to the clinical, enrichment, or entertainment marketplaces requires a more sophisticated understanding of the regulatory environment, a refined approach to optimization, and a move towards testing that achieves the standards of clinical trials (Lu et al. 2016).