Pain, from perception to action: A computational perspective

Summary

Pain is driven by sensation and emotion, and in turn, it motivates decisions and actions. To fully appreciate the multidimensional nature of pain, we formulate the study of pain within a closed-loop framework of sensory-motor prediction. In this closed-loop cycle, prediction plays an important role, as the interaction between prediction and actual sensory experience shapes pain perception and subsequently, action. In this Perspective, we describe the roles of two prominent computational theories—Bayesian inference and reinforcement learning—in modeling adaptive pain behaviors. We show that prediction serves as a common theme between these two theories, and that each of these theories can explain unique aspects of the pain perception-action cycle. We discuss how these computational theories and models can improve our mechanistic understandings of pain-centered processes such as anticipation, attention, placebo hypoalgesia, and pain chronification.

Graphical abstract

Bioinformatics; Neuroscience; Sensory neuroscience

Introduction

Pain is complex and multidimensional. It includes the sensory-discriminative, affective-emotional, and cognitive-evaluative components and is the result of dynamic interactions of multiple central and peripheral neural processes.^1,2 Acute pain serves to protect us from predictable harm. However, when acute pain persists into a chronic phase, it contributes to the dual public health crises of under-treatment and opioid overuse and addiction. A detailed understanding of pain mechanisms is necessary to address these healthcare issues.

In the brain, there is no “pain cortex”. Instead, a distributed network of cortical-subcortical areas contributes to pain processing.^3,4,5 During pain processing, sensory-motor integration occurs, and this integration further interacts with memory to guide decision-making. Pain experiences are subjective and highly variable across subjects, and are dependent on contextual, social, and cultural factors.⁶ Furthermore, pain encompasses both hedonic and motivational aspects of an unpleasant experience.⁷ Pain serves as a “teaching signal” to avoid or minimize predictable harm and to modulate physiological (e.g., homeostatic, autonomic, endocrine, and immune) responses that further affect motivated behaviors (e.g., exploration and social interaction). To fully understand these complex factors, it is imperative to develop a principled computational framework and model individual components.

The mammalian pain system consists of ascending and descending pathways, including the peripheral nerves, spinal cord, and cerebral cortex. To date, multimodal neuroimaging and multisite electrophysiology techniques have been used to uncover pain mechanisms of peripheral and central nervous systems.^8,9,10,11 In contrast to a wealth of experimental data, however, there is a relative lack of computational theories and modeling in pain research.^12,13,14 Several reviews on pain have focused on nociception and negative emotion,^15,16 expectation,¹⁷ motivational decision and action,¹⁸ as well as reinforcement and control.^13,19 Predictive coding and Bayesian models for perception have been widely presented in the literature,^20,21,22,23 but only recently been adopted in pain research.^{12,24,25,26,27,28,29,30,31,32} Gradually, a more general computational framework is also beginning to emerge to accommodate the exponential growth of data in pain research.

In this Perspective, we systematically appraise the role of computational theories and models for studying a wide range of pain-associated processes, with a goal to present a coherent theme that establishes the computational foundation for both theorists and experimentalists in pain research. Specifically, we will focus on theories of predictive coding and reinforcement learning (RL), both of which share a common machinery of “prediction” and “prediction error”. Predictive coding and Bayesian theory offer a predictive inference framework that can be used to understand pain perception across time. Meanwhile, RL provides a mechanistic framework to understanding how the brain supports pain-related actions and decisions.¹³ Together, these two prediction-driven theories form a computational foundation for the “pain perception-action cycle” and goal-oriented decision-making.

Motivation for new computational approaches in pain research

Good theories can not only deepen understanding of biological processes but also generate new predictions to validate current and future experiments.³³ An experimental design may be complex, and data interpretation often requires making some basic assumptions and inferences. On the other hand, modeling also requires information extraction and abstraction (e.g., an internal world model) to identify the common computational principle or mechanism (Figure 1A). There are several reasons to call for new theories and models to study pain. First, computational approaches can complement and overcome some challenges in the current pain experiments, including the lack of quantifiable readout in complex pain behaviors and the tendency to emphasize sensation over action in pain experiences. Furthermore, theory can potentially harmonize the findings across species and modalities, and fill in the knowledge gap in both circuit and behavioral-level understandings. Theories or models may also help constrain experimental variables and guide experimental designs.

Figure 1.

Examples of computational theories and models

(A) Levels of abstraction and update of theory and model via error feedback.

(B) Schematic diagram of predictive coding for pain perception.

(C) The perception-action cycle is intertwined with internal prediction loops, where different stages of prediction can be applied in pain-related processing.

(D) Schematic diagram of reinforcement learning (RL). The agent receives nociceptive input x_k(t) at the k-th trial, updates the value function, selects the action, receives the reward (or punishment), produces a prediction error (PE), which is further used to update the weights and value function.

(E) Schematic diagram of the computational model with arbitration between model-based and model-free learning systems (Wang et al., 2018, ref. ³⁴, figure modified with permission, CCBY 4.0 license).

(F) The predicted response outcome (PRO) model (Alexander and Brown, 2011, ref. ³⁵, figure reproduced with permission, Springer Nature). (i) The model learns predictions of future outcomes (e.g., error or correct feedback) based on task-related cues (S) signaling the onset of a trial is presented. Over the course of a task, the model learns a timed prediction (V) of possible responses and outcomes (r). The temporal difference learning signal (δ) is decomposed into its positive and negative components (ω^P and ω^N, respectively), indicating unpredicted occurrences and unpredicted non-occurrences, respectively. (ii) ω^N accounts for typical effects observed in the PFC from human imaging studies. Conflict and error likelihood panels show activity magnitude aligned on trial onset; error and error unexpectedness panels show activity magnitude aligned on feedback. Model activity (vertical axis) is in arbitrary units. HEL, high error likelihood; LEL, low error likelihood. (iii) Typical time courses for components of the PRO model.

A second reason for studies on theoretical modeling is that a number of older theories on pain, such as the intensity theory, specificity theory, pattern theory, and gate control theory (see ref. ^36,37), have been outdated. For instance, the intensity theory believes that pain results from excessive stimulation of the sense of touch, and defining pain as an emotion that occurs when the stimulus intensity and central summation are greater than the threshold. The specificity theory of pain argues that the summation of pain receptors produces nerve impulses which are transmitted to a pain-specific system in the brain, and views nociception independently of other sensory pathways. The pattern theory of pain hypothesizes that spatiotemporal pattern of impulses from the peripheral nerves encodes the pain sensation and intensity, and these patterns only occur with intense stimulation. The gate control theory proposes that non-nociceptive neurons ascending to the dorsal horn of the spinal cord activate local inhibitory neurons which in turn inhibits smaller-diameter nociceptive pain fibers, thereby halting the transmission of sensory information to the brain.³⁸ However, all of these older theories do not sufficiently account for the integration of active sensing, cognition, and decision-making, and may thus underestimate the importance of dynamic changes in nociceptive/sensory/motor variables in real-world environments. In other words, pain is not only a multisensory experience but also involves at least partially conscious choices and active behavioral responses. A complete understanding of pain requires models that can integrate sensory experience with cognitive choices and motor actions.

Finally, learning and adaptive behaviors are essential to every aspect of pain processing, with examples ranging from pain anticipation, perception, avoidance, recuperation, and chronification. All these adaptive behaviors are driven by learning.^13,19 Pain is a teaching signal. Throughout evolution, animals develop the ability to adapt their bodily or physiological responses to detect and avoid predictable injury and pain. Hence, a new computational framework is urgently needed to incorporate such learning paradigms into the understanding of pain.

Computational framework for studying the pain perception-action cycle

The perception-action cycle is found in all goal-directed behaviors.³⁹ Within this cycle, the brain needs to execute a series of cognitive functions including sensory assessment, attention, memory, and sensorimotor integration. This perception-action cycle can be used as a conceptual model for pain processing.⁴⁰ A pain perception-action cycle has several key components: sensory-motor integration, motivation, and learning. The principle of prediction comprises many aspects in the pain perception-action cycle: predictive coding can be used to explain pain perception and expectation, whereas RL provides conceptual understanding for motivation-driven action. Indeed, prediction is fundamental to numerous pain behaviors including pain anticipation, pain avoidance, fear, pain catastrophizing, chronic pain aversion, and placebo analgesia.^16,41,42,43 A mismatch between prediction and sensory inputs generates a surprise signal representing prediction error (PE), which further drives aversive learning associated with nociception, resulting in both adaptive and maladaptive action. In this section, we will review several computational theories that are essential to prediction and learning for understanding the pain perception-action cycle (Table 1).

Table 1.

Theories and models for understanding the pain perception-action cycle

Theories and Models	Computation	Pain applications
Predictive coding	Prediction of sensory or motor signals	Pain perception, sensorimotor integration
Bayesian theory	Prediction of posterior and posterior mean	Pain perception, placebo analgesia
Reinforcement learning	Prediction of value function	Decision-making, planning, avoidance, pain conditioning, pain chronification
Avoidance/escape learning	Prediction of value function, integration of multi-factors	Pain anticipation/avoidance, pain-related fear conditioning
Associative learning	Modulation of “associability” strength	Attention modulation for pain, fear-conditioned pain modulation, pain extinction
Predicted response-outcome (PRO) model	Action selection and prediction	Pain-associated decision-making
Motivation-Decision model	Weighting the utility function	Decision-making in cost-benefit conflicting or competitive choices

Our main objective of this Perspective article is to educate pain researchers on computational approaches for pain studies. Before reviewing detailed pain applications of computational models, we hope to present a high-level description of theories and models to familiarize pain researchers with basic mathematical backgrounds and necessary technical jargons. Many models are conceptually connected, while hybrid models can also be employed in pain applications. To link these theories with the various pain applications, we aim to provide multiple lines of evidence and neural correlates of PEs in pain circuits by presenting model discussions.

Predictive coding and Bayesian theory: Prediction for sensory perception and motor response to pain

To address uncertainties (i.e., “what”, “when”, and “where”) in a dynamic environment, the brain uses prior information to guide both sensory perception and motor output.^44,45,46 A useful paradigm to describe this process is predictive coding. In this paradigm, sensing and action describe the inversion of an internal generative percept model, which continuously adapts to additional sensory perceptions and/or motor preparation. Such predictive coding paradigms have provided insight into perceptual inference, motor control, and multisensory integration.^21,23,47,48 In this view, sensing is active and predictive, and attention can be used to bias the perceptual selection of multiple sensory inputs.

In predictive coding, the bottom-up (sensory, nociceptive, and proprioceptive) input and top-down (expectation and attention) output are integrated to compute a PE, which is further used as a form of feedback to update the internal model for subsequent prediction (Figure 1B).

Prediction = Input + gain × PE

The PE represents a surprise factor, and the uncertainty of this surprise factor can also be characterized by its variance or the inverse of gain in a system. A small PE and/or large variance would call for a small correction; in contrast, a large PE and small variance would call for a big update. Predictive coding can successfully explain both perception and action, and thus the perception-action cycle consists of two intertwining internal prediction loops (Figure 1C).

Bayesian theory further generalizes predictive coding with a feature of probabilistic inference. In the case of nociception, bottom-up sensory inputs from the periphery produce a likelihood for perceptive and behavioral response, whereas neurons in the brain which undergo plasticity as the results of prior experiences can also provide top-down modulatory outputs to further shape these responses. Take pain perception (including placebo hypoalgesia) as an example, how do we ultimately perceive pain (“pain belief”) does not only depend on how the nociceptive input, sensory input (such as vision), and proprioceptive input signaling alarms (“likelihood of injury or harm”) but also on our attention, expectation, context, and environment (“prior”). Through evolution and experiences, our brains have built empirical internal predictive mapping models for the likelihood of an event to occur. Such directional neural circuit response can be characterized by a generative predictive model (denoted by a probability distribution P(observed | model parameter)), where Bayesian inference can be used to compute the most likely causes of observed neural response (i.e., posterior) by optimally integrating prior experience and observed nociceptive inputs.

$$\text{Posterior}\propto \text{Prior}\times \text{Likelihood}$$

$$\propto \text{Prior}\times \int P\left(\text{observed}|\text{latent}\right)\times P\left(\text{latent}\right)\mathrm{d}\left(\text{latent}\right)$$

From which the pain belief can be inferred from the statistics of posterior distribution (e.g., mean and variance). Importantly, this often forms an inferential loop as the effect of such information may further drive top-down modulation. In this inference model, it is common to introduce a latent variable, which can be either a continuous or discrete random variable, to account for unobserved neural processes (e.g., attention and modulatory input). In the above pain perception example, this is equivalent to averaging out all possibilities of unknowns given their respective priors in the inference. When all the unknowns are equally possible (i.e., noninformative priors), then the posterior computation becomes simplified. In a general case, however, exact inference of the full posterior distribution is computationally intractable (for either computer or brain) and often relies on certain variational approximations, producing different classes of approximate Bayesian inference methods (Box 1). A special case of predictive coding is the Kalman filter, which is an optimal Bayesian estimator for the linear Gaussian system.

Box 1. Bayesian inference.

Let $x$ denote the nociceptive input, and $z$ denote the pain percept. Given the prior expectation $p\left(z\right)$ and the likelihood $p\left(x|z\right)$, the goal of Bayesian inference is to estimate the posterior

$$p\left(z|x\right)=\frac{p\left(z\right)p\left(x|z\right)}{p\left(x\right)}\propto p\left(z\right)p\left(x|z\right)\left(\mathrm{e}\mathrm{x}\mathrm{a}\mathrm{c}\mathrm{t}\mathrm{B}\mathrm{a}\mathrm{y}\mathrm{e}\mathrm{s}\mathrm{i}\mathrm{a}\mathrm{n}\right)$$

In the presence of a latent variable $h$, the posterior estimation needs to integrate over the distribution $p\left(h\right)$. For computational simplicity, $p\left(h\right)$ can be approximated by a Gaussian distribution $N\left(h|{\mu }_h,{\sigma }_h\right)$ with mean ${\mu }_h$ and SD${\sigma }_h$

$$p\left(z|x\right)\propto \int p\left(z\right)p\left(x|z,h\right)p\left(h\right)\mathrm{d}h\approx \int p\left(z\right)p\left(x|z,h\right)N\left(h|{\mu }_h,{\sigma }_h\right)\mathrm{d}h.\left(\mathrm{G}\mathrm{a}\mathrm{u}\mathrm{s}\mathrm{s}\mathrm{i}\mathrm{a}\mathrm{n}\mathrm{a}\mathrm{p}\mathrm{p}\mathrm{r}\mathrm{o}\mathrm{x}\mathrm{i}\mathrm{m}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\right)$$

Alternatively, one can compute the lower bound of the marginal log likelihood $\log p\left(x\right)$ by introducing a variational posterior distribution $q\left(z,h\right)$, such that

$$\log p\left(x\right)=\log \int \mathrm{d}h\int \mathrm{d}zp\left(h\right)p\left(z,x|h\right)$$

$$=\log \int \mathrm{d}h\int \mathrm{d}zq\left(z,h\right)\frac{p\left(h\right)p\left(z,x|h\right)}{q\left(z,h\right)}$$

$$\ge \int \mathrm{d}h\int \mathrm{d}zq\left(z,h\right)\log \frac{p\left(h\right)p\left(z,x|h\right)}{q\left(z,h\right)}\equiv \mathcal{F}\left(q\left(z,h\right)\right)\left(\mathrm{f}\mathrm{r}\mathrm{e}\mathrm{e}\mathrm{e}\mathrm{n}\mathrm{e}\mathrm{r}\mathrm{g}\mathrm{y}\right)$$

where the free energy $\mathcal{F}\left(q\left(z,h\right)\right)$ approximates the lower bound of $\log p\left(x\right)$.

Let’s consider a simple example with the prior $p\left(x\right)\sim N\left({\mu }_1,{\sigma }_1\right)$ and likelihood $p\left(x|z\right)\sim N\left({\mu }_2,{\sigma }_2\right)\text{,}$ then the posterior $p\left(z|x\right)$ is also Gaussian, with the new variance $\frac{{\sigma }_1^2{\sigma }_2^2}{{\sigma }_1^2+{\sigma }_2^2}$ and new mean $\frac{{\sigma }_2^2}{{\sigma }_1^2+{\sigma }_2^2}{\mu }_1+\frac{{\sigma }_1^2}{{\sigma }_1^2+{\sigma }_2^2}{\mu }_2$. Let ${\xi }_1=\frac{1}{{\sigma }_1^2}\text{and}{\xi }_2=\frac{1}{{\sigma }_2^2}$ denote the precision (inverse variance) parameters; then the new precision is $\xi ={\xi }_1+{\xi }_2$, and the new posterior mean estimate is weighted by their respective precision parameters: $\mu =\frac{{\xi }_1}{\xi }{\mu }_1+\frac{{\xi }_2}{\xi }{\mu }_2$. Alternatively, the posterior mean can be rearranged as an incremental update form from the prior: $\mu ={\mu }_1+\frac{{\xi }_2}{\xi }\left({\mu }_2-{\mu }_1\right)$, where the relative precision $\frac{{\xi }_2}{\xi }$ can be viewed as a learning rate, and $\left({\mu }_2-{\mu }_1\right)$ can be viewed as a prediction error (PE).

An effort that combines predictive coding and Bayesian theory is Bayesian predictive coding,²² where the predicted sensory input is equivalent to the mathematical expectation with respect to the posterior of the input

$$\text{Prediction}=\int \text{Input}\times \text{Posterior}\left(\text{input}\right)\mathrm{d}\left(\text{input}\right)$$

$$\propto \iint \text{Input}\times P\left(\text{input}\right)\times P\left(\text{observed}|\text{latent},\text{input}\right)\times P\left(\text{latent}\right)\mathrm{d}\left(\text{input}\right)\mathrm{d}\left(\text{latent}\right)$$

The computation can be understood through a hierarchical inference process.

How do predictive coding and Bayesian theory help our understanding of pain perception? To date, these two theories have been advocated to account for behavioral and neural data in pain processing.^{12,27,49,50,51,52,53} In the context of inference for pain, PE and prediction may be computed in different brain regions. A detailed discussion on neural correlates of these PE signals will be presented in the next section. Additionally, communications of PE and prediction signaling can be manifested in gamma and alpha/beta oscillations in the bottom-up and top-down pathways, respectively; one possibility is within the S1-ACC circuits.⁵⁴ The PE is computed in the higher frequency (such as gamma) oscillations, whereas the prediction is updated in the lower frequency (such as alpha and beta) oscillations, as shown in experimental findings in many pain studies.^27,55,56

Reinforcement learning: Prediction of the pain-aversive value

Motivation forms another key component of the pain perception-action loop. RL is an important tool to understand motivated actions and decision-making.⁵⁷ Through experience-based learning, a biological agent receives an input from the environment and seeks an action in a certain state to maximize the reward and pleasure (such as pain relief) or to minimize the cost. The problem of determining the action that leads to a goal-directed outcome can be formulated as a temporal credit assignment problem. For instance, if a runner has a long-term objective to run a marathon, then a series of planning and action will be made to reach the goal, which ultimately results in a large emotional reward. The same runner will experience physical pain during training; yet the runner is motivated to endure the pain.

Through learning the associations between stimuli (“state”), actions, and the occurrence of pleasant or unpleasant events, RL is capable of predicting future rewards or punishments.⁵⁸ An RL algorithm is model-free if it only replies on real samples from the environment and never uses generated predictions of next state and next reward to alter behavior. One of the most popular model-free RL models is known as temporal difference RL (TDRL) (Box 2 and Figure 1D). The basic idea of the RL model is to formulate a value function and use it to compute the reward PE to guide action selection; the value function is further updated from the reward PE as explained in the following equation.

$$\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{d}\mathrm{i}\mathrm{c}\mathrm{t}\mathrm{e}\mathrm{d}\mathrm{v}\mathrm{a}\mathrm{l}\mathrm{u}\mathrm{e}\mathrm{f}\mathrm{u}\mathrm{n}\mathrm{c}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}=\mathrm{o}\mathrm{l}\mathrm{d}\mathrm{v}\mathrm{a}\mathrm{l}\mathrm{u}\mathrm{e}\mathrm{f}\mathrm{u}\mathrm{n}\mathrm{c}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}+\mathrm{g}\mathrm{a}\mathrm{i}\mathrm{n}x\mathrm{r}\mathrm{e}\mathrm{w}\mathrm{a}\mathrm{r}\mathrm{d}\mathrm{P}\mathrm{E}$$

In this case, the reward PE can be used to increase the propensity to perform actions resulting in higher-than-expected rewards, as well as to update the value function and future predictions. Another model-free RL model is the actor-critic model, which generalizes the TDRL to allow policy iteration, which simultaneously allows action learning (actor) and value learning (critic).⁵⁷ In short, the actor-critic model separates PE from the action selection process: the critic learns to calculate PE, and uses it to predict the value of the current environmental state, and the actor uses the PE to learn to select between competing possible actions. The dorsolateral and ventral striatum have been implicated for the actor and critic roles, respectively.⁵⁹ RL has been successfully adopted in a series of human experiments in the context of pain.^60,61 Specifically, the ventral striatum and the anterior insula encode PE signals predicted by RL models in a fMRI study of higher order aversive conditioning, where humans were instructed to learn a computational strategy to predict pain.⁶⁰ In contrast, the amygdala and midbrain tended to reflect reward-like signals predicted by RL models, where humans learned to generate expectations of pain relief.⁶¹ Detailed discussions of neural correlates of PE signals in distributed pain circuits will be presented in the next section.

Box 2. Temporal difference reinforcement learning.

One of the most popular RL models is the temporal difference RL (TDRL) model.⁵⁷ In the simplest form of TDRL model, the value function $V\left(s\right)$ is associated with a state $s$, and represents the expected value of the temporally discounted sum of all future rewards:

$$V\left(t\right)=E\left[{\gamma }^{0}r\left(t\right)+{\gamma }^{1}r\left(t+1\right)+{\gamma }^{2}r\left(t+2\right)+\cdots \right]$$

where 0 < $\gamma$ ≤ 1 is a discount factor for the reward. At the k-th step (or trial) of the perception-action cycle, the TDRL model is used to update the value function

$$V\left(s_{k+1}\right)\leftarrow V\left(s_k\right)+\alpha \left[r_{k+1}+\gamma V\left(s_{k+1}\right)-V\left(s_k\right)\right]$$

where $r$ denotes the (positive, zero or negative) reward, and 0 < $\alpha$ < 1 is a small learning rate parameter. When $\gamma =0$, the TDRL rule is reduced to the Rescorla-Wagner rule. The error term ${\delta }_k\equiv r_{k+1}+\gamma V\left(s_{k+1}\right)-V\left(s_k\right)$, derived from the difference in predictions over successive time steps, is called the reward PE or temporal difference (TD) error. This error is known to be represented in the midbrain dopamine (DA) neurons, and has been implicated in other cortical processing.

In a TDRL model, learning may occur continuously from moment to moment and is not artificially constrained to specific trial numbers. Algorithmically, the temporal credit assignment problem can be resolved efficiently by the eligibility traces (Figure 1D). In a generalized form of TD($\lambda$) model, the value function is represented by a linear functional approximation between the feature $\mathbf{x}$ and weight $\mathbf{w}$: $V\left(t\right)=\mathbf{w}·\mathbf{x}\left(t\right)$, the weights are updated by the following TDRL rule:

$$\Delta \mathbf{w}\left(t\right)=\alpha \delta \left(t\right)\mathbf{e}\left(t\right)\text{,}\mathbf{e}\left(t\right)=\lambda \mathbf{e}\left(t-1\right)+\mathbf{x}\left(t\right)$$

where 0 < $\lambda <1$  is a decay parameter that determines the plasticity window; a recently active stimulus will have high corresponding eligibility traces, and accommodate for a larger update. When $\lambda =0$, the standard TD(0) learning is recovered.

Although reward and pain represent pleasant and unpleasant experiences, respectively, they can both trigger motivated behaviors through RL models. The nature of reinforcement can be either positive or negative. Positive reinforcement works by introducing a reward to increase the likelihood of a behavior, whereas negative reinforcement learning involves the removal of an unpleasant stimulus (e.g., pain relief).^7,62 In the midbrain mesolimbic dopamine system, dopaminergic (DA) signaling was found in the presence of not only appetitive stimuli (“reward”) but also aversive stimuli (“pain”),^63,64,65,66 where salience-coding DA signaling is similar for pain and reward. On the other hand, DA signaling can represent hedonic value or the sign of the learning signal in reward or pain relief.^62,67 However, reward and pain involve distinct circuit pathways: aversive DA signaling involves ventral tegmental area (VTA)→cortical projections, whereas reward DA signaling involves VTA→ventral striatal or VTA→nucleus accumbens (NAc) projections.⁶⁸ These two systems are dissociable and yet also interconnected.^7,69 Additionally, DA signaling has been found to be important for motivational avoidance behavior; neuroimaging has suggested that dopamine dysfunction plays a role in chronic pain.^70,71,72,73 Chronic pain decreases the activity of VTA DA neurons, reduces motivation for natural rewards, and drives anhedonia-like behavior.⁷⁴ The RL theory may thus provide a useful framework to model DA-related chronic pain and addiction behaviors.^13,75

How can RL theories help explain strategies of value learning and action learning in the context of pain? Pain presents as a strong motivational drive to impact decision-making, and there is also a neurobiological overlap between decision-making and pain.^18,76,77,78 The adaptive pain behaviors in animals and humans may have originated evolutionarily to promote survival by minimizing harm, and may further involve higher order cognitive processes and planning. Therefore, the decision on how to plan an action to escape from or endure potential harm in order to maximize future reward can be formulated as a temporal credit assignment problem.¹³ A number of top-down pathways have been implicated in the control or modulation of pain-related decision-making processes. These include the projections from the prefrontal and midcingulate areas to mid- and hindbrain structures such as the periaqueductal gray (PAG) and NAc,^41,79 as well as the projections to the insula cortex and thalamus.^80,81,82

Associative learning: Modulation of PE and association for pain

The ability to learn associations between cues and pain allows the prediction and avoidance of experiences of pain and injury.⁸³ Operant conditioning (or instrumental conditioning) is a type of associative learning through which the strength of a behavior is modified by reinforcement or punishment. Similar to RL, associative learning uses PE to guide learning; but it also applies a modulating factor (such as attention) to directly or indirectly change the association between a conditioned stimulus (CS) and an unconditioned stimulus (US).⁸⁴ The Rescorla-Wagner model is a special class of associative learning model that uses the signed PE to determine the effectiveness of USs.⁸⁵ Another class of associative learning model employs the unsigned PE to determine the eligibility of CSs.^86,87,88 PE signaling has been found in the amygdala in the context of fear.^89,90

How does associative learning relate to pain? Pain is a primary reinforcer in associative learning in animal models of fear conditioning.¹⁸ During a Pavlovian fear-conditioning paradigm, the expected shock probability of a noxious stimulus and associability (“uncertainty”) were estimated from an associative learning model.⁹¹ Human imaging studies have also found a role of associative learning in pain expectation as well as in placebo responses.^42,92,93 Recently, Zhang and colleagues proposed an associative learning model for tonic pain relief in an escape-learning paradigm. In this study, they found that the uncertainty (“associability”) signal in the associative learning model was encoded in the pregenual anterior cingulate cortex that controls the level of tonic pain aversion, and the error signals in the dorsal putamen explained active seeking of pain relief.⁹⁴ Furthermore, another human neuroimaging study has shown that associative learning signals—the learned associability and PE—are correlated with fMRI responses in amygdala-striatal regions, corresponding to the fear learning circuit.⁹⁵

Avoidance learning and escape learning: Responses to aversive pain stimuli

Adaptive pain behaviors can be explained by various instrumental learning theories driven by reward or punishment. There is a critical difference between reward- and punishment-based learning because of the distinct nature and asymmetry of how the brain process appetitive and aversive outcomes. Specifically, avoidance (or escape) learning is referred to the scenario in which animals or humans can learn a response to avoid (or terminate) experiencing an aversive and painful stimulus. For instance, in avoidance learning, a mouse can avoid the electric shock by moving to the opposite chamber after the CS is presented; in escape learning, a mouse responds to the US instead of the CS by escaping to the opposite chamber. In avoidance learning, the action can be active or passive (i.e., inaction, as action in the presence of warning stimuli has previously led to pain).

How do we develop computational models for avoidance learning behaviors in response to aversive pain stimuli? Avoidance learning can be formulated as a temporal difference model similar to RL,⁹⁶ and can be used to model pain-avoidance conditioning of a non-noxious stimulus.^97,98 Recently, a hybrid model-based and model-free avoidance learning paradigm was proposed for a two-step aversive decision-making task,⁹⁹ in which the avoidance behaviors are controlled by two different systems: a “model-free” habit system and a “model-based” cognitive system. The model-free system bears a form of a specialized RL model, whereas the model-based system learns a forward internal model of state transition; the reliabilities of the two models are evaluated from their respective PEs, and the ultimate value function consists of a weighted sum of the two value functions (V_MF and V_MB) of two systems (Figure 1E). Additionally, Jepma and colleagues used model-free RL to fit the behavioral data of participants performing an instrumental pain-avoidance learning task; they also found that pain PEs were encoded in subcortical and limbic brain regions, whereas no-pain PEs were encoded in frontal and parietal cortical regions.³⁴

What is the neural basis of avoidance learning? The neural substrates of avoidance learning are composed by a WHEN system (limbic circuits, including the amygdala, hippocampus, cingulate cortex, and limbic thalamus) and a WHAT system (motor circuits). Pain and fear are two major factors that influence motivational behaviors: pain motivates recuperative behaviors that promote learning, whereas fear activates defensive behaviors to guard against dangers. These two motivational systems are thought to serve distinct and competitive functions, and as such, a perceptual-defensive-recuperative model has been developed to account for the relationship between fear and pain.¹⁰⁰ The amygdala and striatum are two subcortical circuits primarily involved in avoidance learning^18,101: the amygdala is responsible for learning about aversive outcomes, and the striatum is implicated in reward-related reinforcement of actions to avoid negative outcomes. The brainstem→amygdala and anterior insula→amygdala pathways play an important role in avoidance of punishment.¹⁰² Recent data have also shown that pharmacological manipulations had no effect on the neural encoding PEs, suggesting that pain-avoidance learning is supported by separate threat- and safety-learning system, and that dopamine and endogenous opioids regulate learning from successfully avoided pain.³⁴

Prediction models for action selection and decision-making in conflicts for pain

Another aspect of the pain perception-action cycle is action. Therefore, it is equally important to develop computational models to understand the process of action and its motivation. The anterior cingulate cortex (ACC) is a central hub for pain sensation and action that connects with the somatosensory cortex, parietal cortex, and supplemental motor area (dorsal pathway) as well as the insula, amygdala, and NAc (ventral pathway). The ACC has been implicated in monitoring conflicts and errors, and using the error feedback to guide actions in aversive scenarios.¹⁰³ Specifically, an RL-motivated computational model known as the predicted response-outcome (PRO) model has been proposed for the ACC, as part of the medial prefrontal cortex (mPFC), to predict the action outcome (Figure 1F).³⁵ The PRO model is a generalization of the temporal difference RL model. However, unlike the classic RL model that learns a scalar prediction of the value of the current state, the PRO model learns predictions for multiple possible outcomes. In this case, the model output represents a temporally discounted prediction of various outcomes in proportion to their probability of occurrence, and the aggregates of outcome predictions generated by this model minus the observed outcomes is explained as “negative surprise”, which reflects the probability of an expected non-occurrence, or a “positive surprise”, reflecting the probability of an expected occurrence.^104,105,106 These surprise signals can be used to guide learning and action prediction. For example, Alexander and Brown used PRO and hierarchical error representation models to demonstrate the role of mPFC in action prediction, in which the PE representation and action selection are interleaved with bottom-down and top-down processing.¹⁰⁶

The action in pain perception-action cycle is sometimes driven by conflicting motivations. Pain affects social decision-making in humans. Animals also need to learn prioritized choices to survive when encountering competitive goals (e.g., safety, food, escape, and mating) in the presence of conflicts (e.g., pain, hunger, predator threat, and fight). A utility function of each choice can be introduced to guide decision-making.¹⁰⁷ The “decision utility” describes the usefulness that one perceives and uses for decision-making, which is rooted in the classical behavioral economics theory.¹⁰⁸ Given multiple competitive choices, a Motivation-Decision model of pain can explain how animals and humans learn to reevaluate the cost-benefit conflict to prioritize decisions.^17,109 According to this model, anything that has higher relevance for survival than pain will receive action preference over pain-related reflexes (by inhibiting nociception); therefore, neural circuits that process multiple conflicting motivations will integrate information to provide selective outputs to drive immediate attention. This Motivation-Decision model also allows us to characterize the pain perception-action loop, as it predicts that top-down expectations may either enhance or inhibit nociceptive transmission and further influence pain perception and future behavioral responses.

Neural correlates of prediction errors in pain circuits

The past few decades have witnessed rapid progress and deeper understanding on the neurobiology of pain.^{3,5,110,111,112,113,114,115} Human neuroimaging has enabled us to study “correlational” changes in brain activity or functional connectivity between a large number of brain areas during various pain states (Figure 2A). Examples of brain regions that respond to acute noxious stimuli and that take part in the pathology of chronic pain include the prefrontal cortex (PFC), ACC, midcingulate cortex (MCC), anterior insula, posterior insula (PI), thalamus, basolateral amygdala (BLA), hypothalamus, striatum, PAG, and rostral ventromedial medulla (RVM). Meanwhile, studies in rodent models have enabled us to dissect the causal roles of pain pathways to show how specific neural circuits, such as the ACC, insula cortex, BLA, and prelimbic (PL) cortex (human analog of dorsolateral PFC), play critical roles in pain processing and regulation (Figure 2A and Box 3).

Figure 2.

Brain regions implicated in representations, prediction, and regulation of pain

(A) Top left and bottom left: The lateral and medial views of the brain map show the key nodes of pain matrix in the brainstem, cerebellum, and cerebral cortex. The table on the right summarizes representative studies from human functional imaging studies (∗) and electrophysiology studies in animals (#), or computer simulations (♦).

(B) Illustrations of several causally identified direct pathways in pain regulation.

Box 3. Causal dissection of cortical and subcortical pathways in pain regulation.

Unlike peripheral and spinal nervous systems where nociceptive information flow follows a relatively linear ascending pathway,¹¹⁶ the brain does not have a single distinct region responsible for integration of nociceptive signals. Instead, a distributed network of cortical and subcortical regions collectively processes and integrates nociceptive signals to give rise to the overall pain experience. Several cortico-cortical and cortico-subcortical pathways have been causally identified in rodent pain studies (Figure 2B). In the S1→ACC pathway, chronic pain enhances the cortico-cortical connection, whereas optogenetic modulation of this projection regulates aversive responses to pain.¹¹⁷ In the pathway from mediodorsal (MD) thalamus to ACC, the weakened excitation of ACC neurons to MD inputs causes excitation/inhibition (E/I) imbalance in pain; activating MD inputs elicited pain-related aversion, whereas direct inhibition of subcortically projecting ACC neurons reproduces the same effect.¹¹⁸ In the afferent MCC→PI pathway, the MCC does not mediate acute pain sensation and pain affect, but regulates nociceptive hypersensitivity.¹¹⁹ In the ACC→RVM pathway, direct cortico-spinal modulation by optogenetics causes behavioral pain sensitization, whereas inhibiting the projection induces analgesic effects.¹²⁰ In the PL→NAc pathway, the prefrontal-striatal pathway provides an important regulation to pain.^79,121,122 In the PFC→PAG pathway, neuropathic pain alters excitability and local connectivity of PFC-PAG neurons,¹²³ and optogenetic activation of this pathway produces analgesic and antianxiety effects.¹²⁴ Furthermore, it has been shown that the BLA→PFC→PAG→spinal pathway was critical for development of mechanical and thermal hypersensitivity in neuropathic pain. Nerve injury strengthens synaptic input from the BLA onto inhibitory PL interneurons, whereas the connectivity changes after pain behavior by reducing descending modulation of spinal pain signals.¹²⁵ Finally, the insula→BLA pathway is critical for the formation of empathic pain. The selective activation or inhibition of the insula→BLA pathway strengthens or weakens the intensity of observational pain, respectively.¹²⁶

Neural evidence for brain region activity correlating to the computational models is a strong support for the theoretical models. To date, experimental research has shown that many types of neurons can carry reward PE signals,¹²⁷ starting from dopamine neurons in the VTA and norepinephrine neurons in the locus coeruleus, as well as pyramidal neurons and interneurons from the superior colliculus, frontal cortex, PAG, and the striatum (Figure 2A). For instance, in the primary sensory cortex, motor cortex, and prefrontal cortex, neuronal expressions of PE have been shown to represent the difference between predicted and observed sensory, motor, or executive signals.^21,48 The PE signals can appear in the form of unsigned PEs (i.e., representing the magnitude of an unexpected outcome) or in the form of signed PEs (i.e., the direction of such deviation). Neural “PE” signals can also appear as a generalized form that goes beyond the reward PE or state PE.¹²⁸

Two prominent examples of neural correlates of PEs are found in the amygdala and ACC in animal studies. In one study, McHugh and colleagues showed that aversive prediction signals are found in the mouse BLA.¹²⁹ Specifically, amygdalar responses during fear conditioning evoked by footshock progressively decreased, whereas responses evoked by the auditory cue that predicted footshock concomitantly increased. The unexpected footshock produced larger amygdala responses than expected footshock. The magnitude of the amygdalar response to the footshock predicted behavioral responses in the following day, and the omission of expected footshock led to a decreased amygdala response below baseline, suggesting a negative aversive PE signal. ACC neurons not only can encode pain-related negative affect but they are also responsible for learning underlying recognition of pain-predictive cues and avoidance.¹³⁰ The ACC can also encode PE and surprise signals.^105,131,132 Elston and colleagues reported that error-related feedback integration is associated with increased inputs from the midbrain to the ACC, and the error is predictive of subsequent behavioral adaptation.¹³³

Meanwhile, human neuroimaging studies over the last two decades, meanwhile, have identified the roles of a broad range of brain regions in prediction and PE during pain processing. For example, Talmi et al. reported that reward PEs are also expressed in a ventromedial prefrontal region extending into the orbitofrontal cortex (OFC), ACC, ventral striatum, hippocampus, bilateral insula, and posterior cingulate cortex.¹³⁴ Additionally, Baliki et al. found that the positive activations of NAc in healthy subjects encode the predictive value of pain relief and anticipate its analgesic potential on chronic pain.⁴¹ In an instrumental pain-avoidance task, Roy et al. found that pain PEs are encoded in the human PAG.¹³⁵ Specifically, they found that the ventromedial prefrontal cortex provides an expectation-input to the PAG (Figure 3A), which further relays PE signals to other prefrontal regions important for behavioral regulation (e.g. the dorsomedial prefrontal cortex). Using a probabilistic thermal pain paradigm, Geuter and colleagues found that the anterior insula responses to cued pain stimuli strictly follow the response patterns according to the predictive coding model, whereas the PI directly encodes the stimulus intensity.⁵⁰ Furthermore, the dorsal anterior insula represents pain intensity and expectations, whereas the ventral anterior insula additionally represents the absolute PE.¹³⁶ Recently, using a mixed US (painful heat or loud sound) experimental protocol, Horing and Buchel reported that the anterior insula correlates with unsigned intensity PEs, irrespective of modality (pain vs. sound), indicating an unspecific aversive surprise signal; whereas the dorsal PI encodes the signed intensity PEs, which are modality specific.⁵³ Together, these results from the human insula cortex offer a novel interpretation of aberrant pain processing as disturbed weighting of predictions and PEs. In a follow-up human EEG study, the same research group reported that the alpha-to-beta frequency band activity in the frontal cortical area is associated with the stimulus intensity expectation, followed by a negative modulation of the gamma-band activity by absolute PEs.¹³⁷ Additionally, Ploghaus et al. studied brain activity during different types of thermal pain mismatch and observed that when painful thermal stimulation was unexpected (i.e., positive PE), fMRI BOLD signals were elevated in the hippocampus, superior frontal gyrus (GFs), superior parietal gyrus (GPs), and cerebellum.⁸³ Activation of these brain regions, however, was not found in the mismatch between the expectation and delivery of nonpainful heat stimulation. Furthermore, when painful thermal stimulation was unexpectedly omitted (i.e., negative PE), the BOLD responses decreased in the left GPs and increased in the hippocampus, GFs, and cerebellum. In human EEG recordings, it has been demonstrated that the peak amplitude of laser-evoked potentials (LEPs) reflects the surprise of painful stimulus rather than the intensity of the stimulus, supporting a macroscopic neural correlate of PE.¹³⁸ Together, PE signals appear to be distributed across multiple brain regions, and the differences in local responses can cause overall circuit changes to process both sensory and affective components of pain.

Figure 3.

Computational models for studies of pain

(A) Dynamic causal modeling (DCM) for identifying plausible brain nodes of effective connectivity among value-encoding and aversive PE-encoding regions (Roy et al., 2014, ref. ¹³⁵, figure reprinted with permission, Springer Nature).

(B) Computational models to characterize self-fulfilling prophecies in pain (Jepma et al., 2018, ref. ¹², figure adapted with permission, Springer Nature). (i) RL model for capturing effects of cued-based expectations on pain and confirmation bias on expectation updating. Perceptual inference of pain is jointly determined by nociceptive input, expected pain, and initial belief of the cue. (ii) Predictive coding is visualized by a graphical model and formulated as a linear Gaussian state-space model (parameterized by three variables $\left\{\mu ,{\sigma }_{\psi },{\sigma }_{\epsilon }\right\}$). Each node is a random variable, and the arrow indicates the statistical dependency between random variables, the uncertainty is characterized by a Gaussian distribution with mean and SD parameters. Bayesian inference produces iterative updating of the posterior distribution of pain. (iii) Expectation updating as a function of PE sign. Negative (aversive) and positive (appetitive) PEs represent the lower-than-expected and higher-than-expected pain, respectively. There was a significant main effect of cue type but no significant interaction between PE and cue type.

(C) Bayesian model prediction for placebo hypoalgesia (Büchel et al., 2014, ref. ⁴⁹, figure adapted with permission, Elsevier). The uncertainty of expectation or pain stimulus is characterized by the precision (inverse of variance) parameter. (i) Impact of the precision of prior expectation on posterior prediction in placebo hypoalgesia experiments. Distributions of prior expectation (red), sensory observation (blue), and posterior of perceived pain (green) are shown with respect to the visual analog scale (VAS) rating. (ii) Impact of the precision of sensory input on posterior prediction in placebo hypoalgesia experiments.

(D) Effects of attention increases orientation selectivity (response profile amplitude) and mismatch selectivity (putative PE) for four test conditions. Time 0 denotes the stimulus onset (Smount et al., 2019, ref. ¹³⁹, CC BY license).

Applications of computational models to understand distinct pain experiences

Pain anticipation

Learning about environmental cues that predict noxious stimuli can modify our response to such stimuli. Pain anticipation represents an important example of such learning paradigm. The anticipation of impending pain drives pain catastrophizing and other enhanced aversive responses. The anticipation of a future reward in the form of pain relief, meanwhile, can result in a range of responses from healthy exercises to harmful drug addiction. Human neuroimaging studies have shown that the mPFC and insula cortex may encode pain anticipation.^110,140 These studies further suggest that distinct neural activities may underlie pain and the anticipation of pain.

Prediction and PE provide a natural explanation for pain anticipation. Recently, Iigaya and colleagues proposed that the total value of the reward predictive cue may be the sum of the anticipation of a future reward and the reward value itself.¹⁴¹ Specifically, at the neural circuit level, the authors argued that the ventromedial PFC tracks the value of anticipatory utility, midbrain DA correlates with information that enhances anticipation, whereas the hippocampus mediates the functional coupling between the PFC and midbrain. Thus, the intensity of pain anticipation is mechanistically summed by predictions as well as PE. Furthermore, Story et al. found that pain anticipation may carry a cost in decision-making, and a delayed outcome can motivate individuals to choose an expedited punishment while waiting for the reward.¹⁴² These two theories explain the potential unpleasantness or aversiveness of pain anticipation itself. Evidence from pain studies also suggests that the degree of predictability may modulate the degree of anticipation. For instance, the differences in pain rating and differences in event-related potential responses between high and low-intensity painful stimulations were significantly reduced when the intensity of the noxious stimulus was random and uncertain.¹⁴³

A prior history of pain experiences can drive future expectations, and such expectations can in turn influence both perception and learning. Furthermore, bidirectional interactions between expectations and experiences can result in self-reinforcing behaviors. Both RL and Bayesian theories have been applied to understand how pain anticipation drives behaviors. Jepma and colleagues examined the modulation of sensory perception in a human fMRI study and compared RL and Bayesian inference models while fitting their experimental data (Figure 3B); they argued that the RL model explains pain perception as a mechanistic process involving prediction and error correction, whereas the Bayesian model explains pain perception in terms of probabilistic inference.¹² This interpretation is still debatable since Bayesian predictive coding models can describe prediction and error correction, whereas RL can also be formulated within a Bayesian framework.

Placebo and nocebo effects

Since the experience of pain is influenced by one’s belief and expectation, placebo treatment that has no intrinsic pharmacological effect may produce analgesia by modifying expectation.^144,145 Psychological and neurobiological mechanisms of placebo analgesia have been well studied,¹⁴⁶ and predictive coding models have been adopted for these mechanistic inquiries.¹⁴⁷ One theory for placebo hypoalgesia is the integration of bottom-up nociceptive signals with top-down prior prediction of pain and/or pain relief.⁴⁹ Bayesian model posits that the brain makes inference or prediction by integrating the uncertainty of expectation for pain or pain relief and the uncertainty of nociceptive information. Thus, Büchel et al. showed that the uncertainty of expectations or sensory information is vital for subjective pain perception (Figure 3C).⁴⁹ Application of the Bayesian theory has led to several important discoveries. First, modulatory neurotransmitters including opioid receptor agonists have been shown to influence the precision of expectation to modulate pain perception.^148,149 Second, a disruption of top-down prediction from prefrontal area as a result of neurodegeneration can exacerbate PEs and consequently lead to a stronger influence of ascending nociceptive signals.^150,151,152 The theory of Bayesian integration in placebo hypoalgesia has been tested in a number of human neuroimaging experiments.^50,51,153 Meanwhile, RL theories can also explain persistent pain modulation after the discontinuation of reinforcement, another feature of placebo effects. Indeed, fMRI experiments have shown that analgesic expectations are associated with PE signals in the ventral striatum, whereas the suppression of striatal PEs mediated by the PFC leads to reduced updating of treatment expectancies and decreased extinction of placebo hypoalgesia.¹⁵⁴

Additionally, the view of placebo effects as a form of modification of expectation suggests the possibility to update individual expectations. Thus, positive expectations of pain relief can drive placebo analgesia and the reduction of pain, and these placebo effects further depend on the length of exposure to prior effective interventions.¹⁵⁵ In contrast, nocebo effects are defined as adverse events related to negative expectations and learning processes that are involved in the modulation of the descending pain pathways. Nocebo hyperalgesia, as an opposite os placebo analgesia, is defined by the role that negative expectations of increased pain intensity play in amplifying the overall pain perception.^156,157 Computational models play a similar role in understanding nocebo effects. In a human neuroimaging study, Tinnermann and colleagues showed that the value information (such as the price of medication) increased nocebo effects, and this effect was mediated by neural interactions between the rostral ACC (rACC), PAG, and spinal cord.¹⁵⁸ Furthermore, the rACC deactivation predicted the strength of reported pain increase during nocebo treatment, serving an inhibitory function on the descending pain. Together, these results not only highlight the importance of value representation in higher cortical areas such as the ACC for pain modulation but they also support the role of predictive coding theory in understanding placebo and nocebo effects.

Attention and pain

An important cognitive component of pain is attention.¹⁵⁹ Pain demands attention; at the same time, attention or distraction can also modulate both sensory and affective expression of pain.^160,161 Attention to pain increases pain sensitivity, whereas distraction from pain reduces it.¹⁶² Specifically, when subjects were distracted during painful stimulation, brain areas associated with the affective division of the ACC, OFC, and PAG showed increased activation, whereas the insula, thalamus, and cognitive division of the ACC showed reduced activation.^163,164 Within the predictive coding framework, attention serves as an important variable to promote neural encoding of PEs.¹³⁹ Specifically, attention optimizes the expected precision of predictions and increases the selectivity for neural responses to a relevant stimulus (Figure 3D). Therefore, attention and prediction are two inherently related processes. Conversely, pain can disrupt attentional performance in both healthy subjects and patients with chronic pain.¹⁶⁵ A noxious and salient stimulus can be viewed as an independent attentional target, and it competes with other attentional targets in the prefrontal circuit during bottom-up and top-down interactions of the prefrontal and subcortical regions.^166,167,168 A conflict between attention and pain avoidance demands a reevaluation of cognitive priority to determine behavioral outputs. At the circuit level, if we adopt the conventional view that the PFC leads the main circuit to predict outcomes, then changes in attention, through changes in prefrontal activities, can modify the motivational gain during pain perception and reshape behavioral response. Furthermore, timing is critical for attention, and attentional modulation provides a mechanism whereby even slight changes in neural activity within the prediction-action cycle, if properly placed in a specific temporal context, can result in large behavioral shifts. For example, a recent finding has shown that low-frequency (2 Hz) optogenetic activation of the prelimbic (PL) cortex slightly decreased the basal firing rates of pain-regulatory PFC neurons, but it effectively inhibited pain behaviors of freely behaving rats.¹⁶⁹ Together, these results support the role of PFC as a modifier of attention and in turn a gain controller for the prediction of behavioral outcomes in pain states.¹⁷⁰ Evidence for prefrontal regulation of pain behavioral control also allows the adoption of a broad range of prediction-based computational models, such as differential reinforcement model,¹⁷¹ to unravel the underpinning of attention and pain.

Pain chronification

Acute pain protects us from danger and bodily harm. Chronic pain, on the other hand, represents a pathological state. In many cases, chronic pain continues even when the initial tissue injury has already been repaired. Chronic pain is in general defined by pain lasting longer than three months, and two hallmark features are frequently observed: hypersensitivity to peripheral noxious stimuli of lower intensity or non-noxious stimuli. The pathogenesis of chronic pain has been widely studied at peripheral, spinal, and brain levels.^{172,173,174,175} In animal and human studies, repeated noxious stimulations can result in abnormal neural responses to noxious stimuli or at baseline across cortical (e.g., the somatosensory cortex, insula, PFC, and ACC) and subcortical areas (e.g., the thalamus, amygdala, and NAc). In human fMRI studies, during pain chronification, the brain undergoes a gradual switch from a predominantly somatosensory pain circuit to one dominated by the affective-processing circuit.^70,176,177 Multiple lines of evidence have pointed to the vital role of mesolimbic or cortico-limbic system in the development, amplification, and prediction of chronic pain.^79,178 The interaction of mesolimbic circuitry and prefrontal circuitry determines the condition of pain transition to an emotional state, where the limbic circuits provide modulatory signals for learning.¹⁷⁹ During this process, the negative motivational value of nociception may increase while the value of reward of pain relief may decrease.¹⁸⁰

While molecular and circuit mechanisms of chronic pain have been well studied,^111,172,181 computational theories including Bayesian learning and RL can further contribute to the understanding of chronic pain, particularly the transition from acute to chronic pain.^40,182 A long-established view of pain chronification theory is operant learning,^183,184 which hypothesizes that positive and negative reinforcement of acute pain behaviors leads to the development of chronic pain.^185,186 Apkarian and colleagues proposed that pain chronification is due to the persistence of the memory of pain and/or the inability to extinguish this memory of pain evoked by an initial injury.¹⁸² According to this theory, pain chronification corresponds to a state of continuous learning, in which aversive emotional associations are established with incidental peripheral inputs.^179,182,187 To emphasize the role of maladaptivity in brain circuits, Baliki and Apkarian proposed a theory to view chronic pain as a repeated failure of avoidant behavior, and postulated that disruption of adaptations in a mesolimbic threshold process that gates the transformation of nociceptive activity to conscious pain is viewed as a key factor for chronic pain development.¹⁶ Specifically, they deconstructed chronic pain into four phases: predisposition, injury, transition, and maintenance. While healthy persistent pain can be treated and recovered, chronic pain represents a maladaptive plastic change in the sensory processing pathway that consequently leads to reorganization of the neocortical circuits. Additionally, the transition from acute to chronic pain can cause low DA delivery in the mesolimbic system.^72,73 Since DA signaling contributes to avoidance behavior in the presence of noxious stimuli, a deficit in DA signaling can impair motivated behavior over time.⁶⁶ Neurobiological and behavioral findings can benefit from computational theories and models to explain the chronicity and chronification of pain.

For instance, the prediction of injury and post-injury inference can be formulated as an inference and control question.¹⁹ The optimal control theory, as a special case of RL, can help model the perceived uncontrollability (or unpredictability) and gradual information restriction leading to pathological chronic pain. Therefore, the control-as-inference view may model pain chronification as a series of maladaptive learning (e.g., wrong associations), maladaptive integration (e.g., lesion-induced information loss), and maladaptive priors (e.g., pessimistic belief or reduced subjective controllability) (see ref. ¹⁸⁸ for a review).

During the transition process, a combination of physical events, memory of past events, and emotions can force the brain develop an actor model (i.e. action learning) and a critic model (i.e. value learning) for optimal long-term action selection. Additionally, persistent, maladaptive learning may cause a cascading effect or vicious body-mind circle (e.g., pain→stress→pain). Overtime, the goal-directed action (“Stimulus-Action-Outcome” association) may gradually reduce to a habit (an autonomous “Stimulus-Response” association) that is beyond volitional control. The basal ganglia—the area responsible for habit learning, also plays a role in nociception and pain.¹⁸⁹ Together, these computational models provide potential tools for modeling different putative mechanisms of pain chronification, whereas associative learning may be also used for modeling pain extinction or unlearning unpleasant emotion.^190,191

Phantom pain

Phantom pain is defined as pain that occurs in an area of the body that is no longer present. Brain mechanisms for phantom pain have focused on cortical reorganization,¹⁹² as well as maladaptive plasticity in both peripheral and central nervous systems.¹⁹³ A deeper understanding of such neural plasticity in phantom pain can be characterized by computational models.¹⁹⁴ For instance, Bostrom and colleagues proposed a physiologically plausible computational model to study the maladaptive reorganization of the primary somatosensory cortex (S1).¹⁹⁵ This model assumes that the S1 dynamically self-organizes based on the neural input, and spontaneous activity in the sensory cortex is the result of discrete neuronal noise and coherent activity. Computational simulations for this model showed that both the amount of reorganization and the level of S1 activity during phantom movements were enhanced, suggesting that phantom pain and persistent representation may share a common mechanism driven by an abnormally enhanced spontaneous activity of deafferented peripheral nociceptive channels.

Understanding of the origin of phantom pain has further benefited from computational modeling, especially models of prediction and PE. According to a stochastic entanglement model,¹⁹⁶ a disruption in the “sensory-motor-prediction” loop causes pain perception and sensorimotor circuitry to be pathologically linked and in turn activated despite the lack of a real nociceptive input. This pathologic link creates a misinformation of prediction in terms of the “perceived location” and “imagined motor execution”. This model can explain several features for phantom pain as well as its therapy, including mirror treatment where a subject is placed in front of a mirror and asked to reimagine the missing limb.¹⁹⁷ This model also predicts that mirror therapy can effectively induce reactivation of the original central and peripheral circuitry involved in motor control of the missing body part, resulting in gradual neuroplasticity to disentangle the maladaptive pain-processing circuitry. Finally, phantom limb pain can be explained by maladaptive information integration, where the brain fails to access persistent congruent multisensory information that is required to infer the state-action value function as a result of sensorimotor reorganization, an important factor likely contributing to pathological chronic pain.¹⁷⁴ Such maladaptive learning can be potentially modeled by the actor-critic RL paradigm.

Conclusion

In this Perspective, we have presented a broad range of theoretical and computational models, with the common theme being that PE and prediction play a central role in understanding pain as a closed loop of perception and action. Prediction and learning-driven computational models can explain a broad range of adaptive pain behaviors, including pain anticipation, pain relief as negative reinforcement, placebo and nocebo effects, and chronic pain. Therefore, computational modeling, in combination with neuroimaging in humans and circuit dissection studies in animal models, can help advance our understanding of pain.