A Semiparametric Inverse Reinforcement Learning Approach to Characterize Decision Making for Mental Disorders

Abstract

Major depressive disorder (MDD) is one of the leading causes of disability-adjusted life years. Emerging evidence indicates the presence of reward processing abnormalities in MDD. An important scientific question is whether the abnormalities are due to reduced sensitivity to received rewards or reduced learning ability. Motivated by the probabilistic reward task (PRT) experiment in the EMBARC study, we propose a semiparametric inverse reinforcement learning (RL) approach to characterize the reward-based decision-making of MDD patients. The model assumes that a subject’s decision-making process is updated based on a reward prediction error weighted by the subject-specific learning rate. To account for the fact that one favors a decision leading to a potentially high reward, but this decision process is not necessarily linear, we model reward sensitivity with a non-decreasing and nonlinear function. For inference, we estimate the latter via approximation by I-splines and then maximize the joint conditional log-likelihood. We show that the resulting estimators are consistent and asymptotically normal. Through extensive simulation studies, we demonstrate that under different reward-generating distributions, the semiparametric inverse RL outperforms the parametric inverse RL. We apply the proposed method to EMBARC and find that MDD and control groups have similar learning rates but different reward sensitivity functions. There is strong statistical evidence that reward sensitivity functions have nonlinear forms. Using additional brain imaging data in the same study, we find that both reward sensitivity and learning rate are associated with brain activities in the negative affect circuitry under an emotional conflict task.

1. Introduction

Mental disorders cause immense disability, accounting for 183.9 million disability-adjusted life-years worldwide (Whiteford et al., 2013). However, despite intense clinical and research efforts devoted to developing pharmacological and behavioral treatments for mental disorders over the years, effective treatment strategies remain elusive, and treatment responses are far from adequate across mental disorders (e.g., 30%−50%). Recently, there has been a fast-growing trend to utilize discovered novel biomarkers and behavioral markers, accompanied by sophisticated computational methods (e.g., machine learning or reinforcement learning), to provide a more accurate characterization of mental disorders and more precise prediction of treatment responses than traditional approaches (Passos et al., 2019).

To address substantial between-patient heterogeneity and limitations in the traditional symptom-based diagnosis, the paradigm shift put forward by the NIMH strategic plan on research domain criteria (RDoC; Insel et al., 2010) initiative calls for re-defining mental disorders by latent constructs identified from biological and behavioral measures across different domains of functioning (e.g., positive/negative affect, social processing, decision making) at different levels (e.g., genes, cells, and circuits), instead of solely relying on clinical symptoms or the diagnostic and statistical manual of mental disorders (APA, 2013). The vision on precision psychiatry (Insel et al., 2010; Williams, 2016) is that research needs to account for extensive diagnostic heterogeneity and substantial between-patient variation in biological, behavioral, and clinical manifestations of disease, examine shared latent constructs across disorders, and match treatments with the underlying pathophysiology.

Learning and decision-making are considered important in the etiology of the mental function (Kendler et al., 1999). An individual’s learning ability and decision-making process may be altered by mental disorders (Pizzagalli et al., 2005). In turn, poor decision-making may lead individuals into high-risk scenarios and experience more mental illness (Kendler et al., 1999). For example, impairment in processing rewards is implicated in various mental disorders. Anhedonia (i.e., inability to experience pleasure) is one of the core constructs of major depression. Pizzagalli et al. (2005) showed that anhedonia may alter how MDD patients process reward in a PRT experiment. Briefly, subjects were presented with two types of stimuli in a computer-based experiment (probabilistic reward task), and one stimulant was rewarded more frequently than the other (details in Section 1.1). Over time, healthy subjects will learn which reward has a higher value and choose the correct stimuli with a higher probability. It is hypothesized that anhedonia may reduce a patient’s ability to learn which reward is more valuable or reduce a subject’s tendency to choose the richer reward. Decision-making models are proposed to evaluate whether these hypotheses are supported (Huys et al., 2013). As another example, restrictive eating behavior in anorexia nervosa can be measured via the Food Choice Task, a computer-based assessment of ratings and choices of food items, and patients’ choices in the task predicts future caloric intake (Foerde et al., 2015).

A useful computational model for describing a patient’s learning behavior and decision-making ability is Reinforcement Learning (RL; Sutton and Barto, 2018). To use RL to describe behaviors or actions generated from a probabilistic reward task, a simple prediction error learning is invoked to characterize the value of different choices depending on key behavioral phenotype parameters, including reward sensitivity and learning rate. Various empirical studies and a meta-analysis have shown that objective, quantitative measures derived from behavioral data can be associated across multiple mental disorders (Pike and Robinson, 2022), making these measures ideal candidate behavioral phenotypes under the RDoC framework. In contrast to regular RL, where one aims to estimate an optimal policy that maximizes the long-term reward, the goal of analysis in behavioral phenotyping in mental health is behavioral cloning or imitation learning (Ross and Bagnell, 2010). That is, given observed patients’ choices on probabilistic reward tasks, how to infer their reward learning ability and value computation process, and how these abilities and processes relate to MDD and, ultimately, are affected by MDD treatments.

There is a gap in using existing imitation learning or inverse RL methods to address the unique challenges presented by mental disorders. For example, the models proposed in Huys et al. (2013) assume that a patient’s decision-making process relies on a function with restrictive parametric linear form, which may oversimplify the complex decision process and interactions between core mental functions. For example, similar to other psychometric models (Gravetter and Forzano, 2018) and as we demonstrate in Section 4, there is often a floor and ceiling effect on the relationship between action and expected reward, which is not captured by a linear model. Some existing imitation learning and inverse RL methods are more flexible to allow nonlinear models (e.g., Arora and Doshi, 2021), but they do not provide important behavioral phenotypes such as learning rate and reward sensitivity and do not borrow information from multiple subjects under a random effects model.

In this work, motivated by the experiments in Pizzagalli et al. (2005) and Trivedi et al. (2016), we propose a semiparametric inverse reinforcement learning approach to characterize reward-based decision-making for patients with mental disorders. Our method incorporates subject-specific learning rates and reward sensitivity as random effects to address between-patient heterogeneity. To borrow information while remaining flexible, we assume that the function that links action to the contrast in expected reward is nonparametric but shared across the patients. We describe the behavioral phenotyping experiment and motivating study in the next section.

1.1. Probabilistic Reward Task and the Motivating Study

The probabilistic reward task (PRT; Pizzagalli et al., 2005) is a computer-based experiment that measures the subject’s ability to modify behavior in response to rewards. On each trial of the task, the subject sees a cartoon face with a short or long mouth (two stimuli). The task is to indicate whether a short or long mouth was presented by pressing buttons ‘C’ or ‘M’ on keyboards. Critically, the correct response to the short mouth was rewarded more frequently than a correct response to the other (i.e., the short mouth is the rich reward), and the difference in length between the short and long mouths is minimal. Participants were given verbal instructions about the task and told that the goal of the task was to win as much money as possible. It is critical that the subject understands that not all correct responses will be rewarded. To maximize the reward, participants should press the correct button, regardless of which face is associated with the higher reward. However, the difference in size between the short and long mouths is designed to be small. Consequently, participants frequently encounter difficulties in accurately perceiving the presented state, which results in a tendency to prioritize states with higher rewards rather than those that are more accurate. Thus, the PRT experiment probes a subject’s ability to learn rich versus lean reward.

Emerging evidence indicates the presence of reward processing abnormalities in MDD (Whitton et al., 2015). MDD patients are more likely to have lower reward learning ability than patients not diagnosed with MDD (Vrieze et al., 2013). There has been significant interest in learning how MDD affects patients’ decision-making in reward learning tasks. In our motivating study, the Establishing Moderators and Biosignatures of Antidepressant Response for Clinical Care (EMBARC) trial, each PRT session consisted of 200 trials, divided into two blocks of 100 trials, with blocks separated by a 30-second break. For each block, 40 correct trials were followed by reward feedback. Correct identification of the short mouth was associated with three times more positive feedback (30 out of 40) than correct identification of the long mouth (10 out of 40). However, subjects were not told about the different frequencies of rewards across the two response types. Figure 1(a) shows the PRT schematic diagram.

Fig. 1.

Schematic of a PRT experiment and EMBARC study design

EMBARC study is designed to identify the difference in reward learning abilities between the patients with MDD and the control group (not diagnosed with MDD) and between SSRI antidepressant sertraline (SERT) and placebo (PBO) in a randomized trial for patients with MDD. Figure 1(b) shows the experimental design diagram. In the PRT experiments, there were 40 subjects from the control group and 168 subjects from the MDD group; within the MDD group, 82 participants were randomly assigned to the SERT group, and the other 86 participants were assigned to the PBO group. Each subject had two PRT sessions, one session at the baseline before treatment (week 0) and one session after one week of treatment (week 1). Note that subjects in the control group also took two sessions in week 0 and week 1. In week 1, five sessions were missing for the MDD group and one for the control group.

2. Methods

2.1. Data from a behavioral task

Consider $n$ subjects from a population. Data consists of a time-series of subjects’ decision-making in a behavioral task (e.g., probabilistic reward task), i.e., ${\left\{S_{it},A_{it},R_{it}\right\}}_{t=1,\dots ,T}^{i=1,\dots ,n}$, where $S_{it},A_{it}$, and $R_{it}$ are the state, action, and reward at time $t$ for the $i$-th subject. In this paper, we assume that there are $m$ states in the state space (i.e. $S_{it}\in \{0,\dots ,m-1\}$) and two actions in the action space (i.e. $A_{it}\in \left\{\mathrm{0,1}\right\}$). Denote ${\mathscr{H}}_{it}={\left\{S_{ij},A_{ij},R_{ij}\right\}}_{j=1,\dots ,t-1}$ as the observed history for the $i$-th subject by time $t$. The reward $R_{it}$ is generated based on $S_{it},A_{it}$, and ${\mathscr{H}}_{it}$ (for some cases, only based on $S_{it}$ and $\left.A_{it}\right).R_{it}$ is assumed to be bounded (either discrete or continuous), and without loss of generality, we assume $R_{it}$ is between 0 and 1. The probabilistic reward task introduced in Section 1.1 is a special case with two possible states. We let $S_{it}=0$ correspond to the state with lean rewards and $S_{it}=1$ correspond to the state with rich rewards. The reward generating distributions conditional on $S_{it}=A_{it}=0$ and $S_{it}=A_{it}=1$ (and history ${\mathscr{H}}_{it}$) are Bernoulli distributions with probability $0<P_{00}<P_{11}<1$. No reward is provided if the $i$th patient does not correctly respond to the state at time $t$.

2.2. Semiparametric inverse RL model

RL is a computational tool used to model patients’ responses to stimuli in behavioral tasks such as PRT. A core concept in RL modeling of PRT is the reward prediction error, the difference between the expected reward and the reward that was received. Mathematically, the reward prediction error is $R_{it}-Q_{it}(a,s)$, where $Q_{it}(a,s)$ is the expected reward or value of taking action $a$ at state $s$ for the $i$th subject at the $t$th trial. This is the foundation of building more complex models to explain patients’ behavioral responses. Specifically, for MDD patients, a failure to adequately process reward is hypothesized to occur via two distinct mechanisms: a reduced sensitivity to received reward or the reduced ability to learn (Huys et al., 2013). The former mechanism is assessed by a parameter referred to as reward sensitivity (denoted by ${\rho }_i$), and the latter is assessed by learning rate (denoted by ${\beta }_i$). When accounting for reward sensitivity, a subject’s reward prediction error is

$${\delta }_{it}={\rho }_i{R}_{it}-Q_{it}(a,s),$$

where the obtained reward $R_{it}$ is discounted by a factor of ${\rho }_i$. Each patient’s value (denoted as $Q_{i,t+1}(a,s)$) naturally evolves over time depending on the past history of value and is updated based on a weighted reward prediction error as

$$Q_{i,t+1}(a,s)=Q_{it}(a,s)+{\beta }_i{\delta }_{it}{I}_{it}(a,s)$$ (1)

where ${\beta }_i\in \left(\mathrm{0,1}\right)$ is a subject-specific learning rate that describes how fast the value of a state-action pair is updated, $I_{it}(a,s)=I\left(A_{it}=a,S_{it}=s\right)$ is the indicator function of the event $\left\{A_{it}=a\right.$ and $\left.S_{it}=s\right\}$. The larger ${\rho }_i$ is, the more a subject’s action will depend on rewards. Equivalently, the expected reward at $t+1$ is the weighted sum of the obtained reward and the expected reward at $t$, i.e., $Q_{i,t+1}(a,s)=\left(1-{\beta }_i{I}_{it}(a,s)\right)Q_{it}(a,s)+{\beta }_i{\rho }_i{R}_{it}{I}_{it}(a,s)$. As the learning rate ${\beta }_i$ approaches one, learning is so fast that the $Q$ values are simply the last experienced outcome. As a note, (1) is also known as the Rescorla-Wagner equation (Rescorla, 1972) in classical conditioning. Figure 2 shows a graphical representation of the model where the green paths show the RL component of updating $Q_{it}$ over trials.

Fig. 2.

Diagram of the semiparametric inverse RL model over trials $t$, where $f$ is the reward sensitivity function, ${\beta }_i$ is the learning rate, and ${\rho }_i$ is related to the relative reward sensitivity.

At time $t=1$, we assume $Q_{i1}(a,s)={\alpha }_{as}$. Define ${\Delta }_{it}={\Delta }_{it}(a,s)$ as the total counts that subject $i$ takes action $a$ at state $s$ before time $t$. Define time ${\tau }_{ik}={\tau }_{ik}(a,s)$ as the $k$-th time such that subject $i$ takes action $a$ at state $s$. According to (1), we can express $Q_{it}(a,s)$ by

$$Q_{it}(a,s)={\left(1-{\beta }_i\right)}^{{\Delta }_{it}}{\alpha }_{as}+{\beta }_i{\rho }_i\sum _{k=1}^{{\Delta }_{it}}{\left(1-{\beta }_i\right)}^{{\Delta }_{it}-k}{R}_{i,{\tau }_{ik}}\text{.}$$ (2)

Clearly, $0\le Q_{it}(a,s)\le \text{max}\left({\alpha }_{as},{\rho }_i\right)$ and $Q_{it}(a,s)$ is also a function of ${\beta }_{i,}{\rho }_i,{\alpha }_{as}$, and history ${\mathscr{H}}_{it}$. Furthermore, $Q_{it}\left(a,s,{\beta }_i,c{\rho }_i,c{\alpha }_{as};{\mathscr{H}}_{it}\right)=c{Q}_{it}\left(a,s,{\beta }_i,{\rho }_i,{\alpha }_{as};{\mathscr{H}}_{it}\right)$. By trial $t$, participants have the updated reward value $Q_{it}(a,s)$ based on $(t-1)$ trials. However, because participants cannot be sure which stimulus would be presented at trial $t$, their action at trial $t$ is based on a “belief” of the expected reward, $W_{it}(a,s)$, defined as a mixture of ${\left\{Q_{it}(a,s)\right\}}_{s=0,\dots ,m-1}$

$$W_{it}(a,s)={\omega }_{ss}{Q}_{it}(a,s)+\sum _{r\ne s}{\omega }_{sr}{Q}_{it}(a,r),$$ (3)

where the parameters ${\omega }_{sr}\in \left[\mathrm{0,1}\right]$ satisfy $\sum _{r=0}^{m-1}{\omega }_{sr}=1$, and they reflect the weights from $Q$-value from different states than $s$.

It is worth noting that the formulation of $Q_{it}(a,s)$ and ${ }^{W_{it}}(a,s)$ are motivated by Huys et al. (2013). We generalize Huys et al. (2013) to allow a nonlinear reward sensitivity function, more than two states (i.e., $S_{it}=1,\cdots ,m$), and use a random effects model to borrow information across subjects to estimate reward sensitivities and learning rates. Specifically, given the believed reward $W_{it}(a,s)$ and the actual stimulus $S_{it}$ at trial $t$, we define a contrast between two actions as

$$Z_{it}=W_{it}\left(1,S_{it}\right)-W_{it}\left(0,S_{it}\right),$$ (4)

and we assume that the action $A_{it}$, conditional on history ${\mathscr{H}}_{it}$ and $S_{it}$, follows the model

$$\text{logit}P\left(A_{it}=1\mid S_{it},{\mathscr{H}}_{it}\right)=f\left(Z_{it}\right)\text{,}$$ (5)

where $f(\cdot )$ is an unknown non-decreasing function satisfying $f\left(0\right)=0$. In other words, the participant is more likely to choose the action whose expected reward is higher, but this probability can be a nonlinear function of the contrast. As a result, we refer to $f(\cdot )$ as a reward sensitivity function. The nonlinear modeling of the decision process may more accurately reveal when subjects are more sensitive to reward under a probabilistic reward task. Finally, to incorporate the heterogeneity among the participants, we assume $v_i=\text{log}\left({\beta }_i/\left(1-{\beta }_i\right)\right),{\gamma }_i=\text{log}\left({\rho }_i\right)$, where $v_i$ and ${\gamma }_i$ are random effects generated from a bivariate normal distribution, i.e., $\left(v_i,{\gamma }_i\right)\stackrel{iid}{~}N(\mu ,\Sigma )$ with $\mathit{\mu }={\left({\mu }_v,{\mu }_{\gamma }\right)}^{\top }$. Note that since $Q_{it}$ is invariant if its initial value and ${\rho }_i$ are scaled by a constant factor $c,f\left(x\right)$ and $f(x/c)$ yield the same action model. Thus, to ensure identifiability, we let ${\mu }_{\gamma }\equiv 1$. By this normalization, ${\gamma }_i$ serves as an indicator for the relative sensitivity of the $i$-th subject $\left(i.e.\left({\gamma }_i-{\mu }_{\gamma }\right)/{\mu }_{\gamma }\right)$. We refer to ${\gamma }_i$ as the relative sensitivity in the rest of the paper. The decision-making model (5) is represented by the blue paths in Figure 2. The rewards are generated based on a participant’s state, choice, and history depending on the experiment paradigm (pink paths in Figure 2).

2.3. Implementation

Under the proposed semiparametric inverse RL/imitation learning model, since $R_{it}$ and $S_{\text{it}}$ given ${\mathscr{H}}_{it}$ are independent of the random effects, the log-likelihood function for subject $i$ is, up to a constant,

$${\ell }_i(\mathit{\mu },\mathbf{\Sigma },\mathit{\alpha },\mathbf{\Omega },f)=\text{log}\left[\iint \left\{\prod _{t=1}^T{P}_{it}\left(v_i,{\gamma }_i,\mathit{\alpha },\mathbf{\Omega },f;{\mathscr{H}}_{it}\right)\right\}\varphi \left(v_i,{\gamma }_i\mid \mathit{\mu },\mathbf{\Sigma }\right)d{v}_{i}d{\gamma }_i\right],$$ (6)

where $\varphi (\cdot ,\cdot \mid \mathit{\mu },\mathit{\Sigma })$ denotes the density of $N(\mathit{\mu },\mathbf{\Sigma })$,

$$\mathit{\alpha }={\left({\alpha }_{as}\right)}_{a=\mathrm{0,1}}^{s=0,\dots ,m-1},\mathbf{\Omega }={\left({\omega }_{sr}\right)}_{s,r=0,\dots ,m-1}$$

$$P_{it}\left(v_i,{\gamma }_i,\mathit{\alpha },\mathbf{\Omega },f;{\mathscr{H}}_{it}\right)=\frac{\text{exp}\left\{A_{it}f\left(Z_{it}\right)\right\}}{1+\text{exp}\left\{f\left(Z_{it}\right)\right\}}.,$$

We maximize

$$ℒ(\mu ,\Sigma ,\alpha ,\Omega ,f)=\sum _{i=1}^n{\ell }_i(\mu ,\Sigma ,\alpha ,\Omega ,f),$$ (7)

to estimate all the parameters. To fit a non-decreasing function $f\left(x\right)$ for $x\in [L,U](L<0,U>0$) where $L$ and $U$ are two pre-given lower and upper bounds for the contrast, we approximate $f$ using a family of monotone splines called I-splines (Ramsay, 1988). Denote $\mathit{M}=\left(M_1,\dots ,M_K\right)$ as a set of M-splines, $\mathit{I}=\left(I_1,\dots ,I_K\right)$ as the corresponding I-splines, where each $I_k\left(x\right)$ is a integral of $M_k(\cdot )$ from $L$ to $x$. Because $M_k(\cdot )$ is non-negative for all $k=1,\dots ,K,I_k(\cdot )$ is guaranteed to be non-decreasing. More details of the representation of M-splines and I-splines can be found in Appendix A.1. We can approximate $f(\cdot )$ by $\tilde{f}\left(x\right)=a+{\mathit{I}}^{\top }\left(x\right)\mathit{b}$. Giving restrictions on the coefficients $b_k\ge 0$ for $k=1,\dots ,K$ enforces $\tilde{f}(\cdot )$ to be non-decreasing. We also need $a=-{\mathit{I}}^{\top }\left(0\right)\mathit{b}$ to ensure $\tilde{f}\left(0\right)=0$. Besides, the first order derivative of $\tilde{f}$ is: ${\tilde{f}}^{\prime }\left(x\right)={\mathit{M}}^{\top }\left(x\right)\mathit{b}$.

We evaluate the double integral using bivariate Gauss–Hermite quadrature (Jäckel, 2005). In the optimization step, because $\mathit{\alpha }$ and $\mathbf{\Omega }$ are linearly constrained, the coefficients of the I-splines are non-negative, so the constrained optimization (Lange et al., 1999) is used to maximize (7). In our numerical studies, the L-BFGS-B algorithm is implemented by the “optim” function in R package “stats” for the case $m=2$ (i.e., two states), when $m>2$, an adaptive barrier algorithm implemented by the “ constrOptim” function in R package “stats” can be used instead. Algorithm 1 presents a detailed version of the semiparametric RL algorithm for parameters and nonparametric function optimization. Note that evaluating the value of joint conditional log-likelihood function (7) is time-consuming, so to accelerate the computation, we evaluate the log-likelihood function for each subject (6) in parallel and sum up each part to obtain (7). The maximum likelihood estimation is obtained when the L-BFGS-B algorithm converges.

Furthermore, denote $\mathbf{\Lambda }=(\mathit{\mu },\mathbf{\Sigma },\mathit{\alpha },\mathbf{\Omega },\hat{f})$ as the maximum likelihood estimate of (7). The subject-specific learning rate and relative sensitivity can be estimated by plugging in $\mathbf{\Lambda }$ into the posterior mean of ${\left(v_i,{\gamma }_i\right)}^{\top }$.

$$\text{E}\left({\left(v_i,{\gamma }_i\right)}^{\top }\mid \mathbf{\Lambda };{\mathscr{H}}_{i,T+1}\right)=\left[\iint {\left(v_i,{\gamma }_i\right)}^{\top }\left\{\prod _{t=1}^T{\text{P}}_{it}\left(v_i,{\gamma }_i,\mathit{\alpha },\mathbf{\Omega },\hat{f};{\mathscr{H}}_{it}\right)\right\}\varphi \left(v_i,{\gamma }_i\mid \mathit{\mu },\mathbf{\Sigma }\right)d{v}_{i}d{\gamma }_i\right]\times \text{exp}\left\{-{\ell }_i(\mathit{\mu },\mathbf{\Sigma },\mathit{\alpha },\mathbf{\Omega },\hat{f})\right\}.$$ (8)

In the Appendix, we study the theoretical properties of the MLEs of parameters and monotone function $f(\cdot )$ and show consistency and asymptotic normality. Furthermore, nonparametric bootstrap is used for inference. Because asymptotic normality holds, as we show in Appendix A.2, we construct 95% bootstrap confidence intervals under the normal distribution (i.e., $MLE\pm 1.96\times \mathit{\text{bootstrap se}}$) in simulation studies and data application. Because the distributions of ${\sigma }_{v,v}^2$ and ${\sigma }_{\gamma ,\gamma }^2$ are highly skewed to the right, we transform them by logarithm before using normal approximation to construct confidence intervals.

3. Simulation Studies

We conducted extensive simulation studies to assess the finite sample performance of the proposed method. To mimic the real study, denote the distribution for generating reward $R_{it}$ conditional on $A_{it}=a$ and $S_{it}=s$ as $p_{as}\left(R_{it}\right)$. We considered $m=2$ (two states), We further let ${\alpha }_{00}={\alpha }_{11}:=\alpha$ and ${\alpha }_{01}={\alpha }_{10}:=0$, and $\mathbf{\Omega }$ satisfy ${\omega }_{00}={\omega }_{11}:=\omega$. As the result, (4) became

$$Z_{it}=\left\{\begin{array}{lll}\omega \cdot Q_{it}\left(\mathrm{1,1}\right)-(1-\omega )\cdot Q_{it}\left(\mathrm{0,0}\right)& \text{if}& S_{it}=1\\ -\omega \cdot Q_{it}\left(\mathrm{0,0}\right)+(1-\omega )\cdot Q_{it}\left(\mathrm{1,1}\right)& \text{if}& S_{it}=0\end{array}\right.$$ (9)

Furthermore, we considered two reward-generating distributions: (Case I—Bernoulli distribution) $p_{11}=\text{Bern}\left(0.75\right)$ (rich reward), $p_{00}=\text{Bern}\left(0.3\right)$ (lean reward); (Case II—Beta distribution) $p_{11}~\text{Beta}\left(\mathrm{3,1}\right)$ (rich reward), $p_{00}~\text{Beta}\left(\mathrm{1,3}\right)$ (lean reward). The rewards at different time points were independently assigned. The states were generated by flipping a fair coin. For the $i$th subject, the transformed learning rate $v_i$ and reward sensitivity ${\gamma }_i$ were generated from a bivariate normal distribution with ${\mu }_v=-2.5,{\mu }_{\gamma }=1,{\Sigma }_{v,v}=0.25,{\Sigma }_{\gamma ,\gamma }=0.16$, and ${\Sigma }_{v,\gamma }=-0.1$. We set $\omega =0.8$ and $\alpha =2$. The nonparametric function was $f\left(x\right)=\text{sgn}\left(x\right)\text{log}\left(2\right|x|+1)$. The I-splines with order $r=2$, internal knots $\{0,\pm 0.6,\pm 1.2,\pm 2.4\}$, and boundary knots ±6 were used in all scenarios.

We generated 200 replicates with sample sizes $n\in \left\{\mathrm{100,250}\right\}$ and number of trials $T\in \left\{\mathrm{100,500}\right\}$. We maximized the proposed joint log-likelihood function (7) by applying a 3 × 3 bivariate Gauss-Hermite quadrature. We used 50 bootstrap samples for inference. In the Supplementary Materials Section S.1.2, we show that using a 3 × 3 Gauss-Hermite quadrature and 50 bootstrap samples are sufficient to provide accurate point estimates and confidence intervals. According to Figure S1 in (Supplementary materials Section S.1.3), most $Z_{it}(i=1,\dots ,n,t=1,\dots ,T)$ lie in (−1.5, 2.5) for $T=100$, hence we evaluate the estimation performance of $f(\cdot )$ at {± 0.5, ± 1, 1.5, 2}.

The simulation results of the parameter estimate from 200 replicates were given in Table 1. The simulation results of the nonparametric function were given in Table 2. For both tables, the relative bias (RB), standard deviation (SD), average bootstrap standard error (SE), and coverage probability of the 95% bootstrap confidence intervals (CP) were reported. Furthermore, we compared our method (Semiparametric RL) with the linear model that $f\left(x\right)=cx$ (Linear RL) in Table 1 and Table 2. Table 1 shows that estimates of group mean ${\mu }_v$ for both semiparametric RL and Linear RL models have small relative biases. It suggests we can estimate the group learning rate with high precision even when the reward sensitivity function $f(\cdot )$ is incorrectly specified. However, the estimates of covariance matrix $\mathbf{\Sigma }$ have much larger relative bias when the reward sensitivity function $f(\cdot )$ is incorrectly specified. Meanwhile, for the semiparametric RL model, the relative bias of all parameters decreases when $T$ and $n$ increase. In contrast, for the linear RL model, the relative bias for all parameters except the group learning rate does not change much as $T$ and $n$ increase. Table 2 suggests that the semiparametric RL estimates of nonparametric function have much smaller relative bias and larger standard deviation than the Linear RL model estimates. Note that the coverage probabilities of semiparametric RL estimates are close to the nominal level (95%). In contrast, the coverage probabilities of Linear RL estimates should be much less than the nominal level. Hence, we conclude that the Linear RL model cannot provide reliable statistical inference on $f(\cdot )$ when the underlying true reward sensitivity function is nonlinear. In the Supplementary Materials Section S.1.1, we also investigated the case when the underlying true reward sensitivity function is linear. We find that the two methods provide estimates with similar relative bias; our Semiparametric RL has a relatively larger standard deviation compared to Linear RL.

Table 1.

Summary of the parameter estimates in 200 simulations with Bernoulli (Case I) and Beta (Case II) reward distribution.

			Case I						Case II
			Semiparametric				Linear		Semiparametric				Linear
$\mathit{T}$	$\mathit{n}$		RB	SD	SE	CP	RB	SD	RB	SD	SE	CP	RB	SD
100	100	${\mu }_v$	0.014	0.316	0.346	97	0.103	0.301	0.055	0.501	0.516	97	0.139	0.385
		${\sigma }_{v,v}^2$	−0.119	0.208	0.301	98	−0.577	0.235	−0.048	0.205	0.390	98	−0.245	0.225
		${\sigma }_{\gamma ,\gamma }^2$	−0.154	0.132	0.132	98	0.533	0.037	−0.100	0.161	0.150	97	0.581	0.037
		${\sigma }_{v,\gamma }^2$	0.163	0.119	0.135	98	−0.251	0.070	0.046	0.137	0.149	97	−0.259	0.069
		$\alpha$	−0.053	0.454	0.445	95	−0.055	0.233	−0.077	0.570	0.529	97	−0.085	0.214
		$\omega$	−0.011	0.052	0.057	96	−0.062	0.061	−0.017	0.077	0.077	97	−0.036	0.080
100	250	${\mu }_v$	0.002	0.169	0.202	97	0.091	0.162	0.014	0.273	0.324	95	0.128	0.240
		${\sigma }_{v,v}^2$	−0.009	0.111	0.126	99	−0.509	0.144	0.077	0.129	0.158	99	−0.166	0.132
		${\sigma }_{\gamma ,\gamma }^2$	−0.070	0.071	0.077	96	0.577	0.019	−0.052	0.106	0.086	91	0.561	0.025
		${\sigma }_{v,\gamma }^2$	0.014	0.061	0.072	98	−0.333	0.040	−0.115	0.075	0.077	97	−0.267	0.041
		$\alpha$	−0.009	0.236	0.279	98	−0.044	0.133	0.004	0.269	0.316	96	−0.061	0.143
		$\omega$	−0.009	0.035	0.036	97	−0.064	0.036	−0.022	0.053	0.055	98	−0.044	0.052
500	00	${\mu }_v$	0.001	0.091	0.102	97	−0.002	0.109	−0.003	0.114	0.128	96	0.042	0.114
		${\sigma }_{v,v}^2$	0.100	0.070	0.076	98	−0.283	0.133	0.139	0.101	0.115	99	−0.168	0.153
		${\sigma }_{\gamma ,\gamma }^2$	024	0.045	0.050	95	0.685	0.009	0.096	0.055	0.058	94	0.689	0.010
		${\sigma }_{v,\gamma }^2$	0.059	0.041	0.050	95	−0.382	0.034	−0.028	0.049	0.053	96	−0.343	0.035
		$\alpha$	0.002	0.220	0.232	95	−0.075	0.135	−0.024	0.243	0.281	98	−0.092	0.133
		$\omega$	0.000	0.007	0.007	95	−0.081	0.011	−0.004	0.014	0.016	99	−0.067	0.010
500	250	${\mu }_v$	−0.001	0.062	0.063	95	−0.003	0.070	−0.003	0.070	0.079	95	0.044	0.080
		${\sigma }_{v,v}^2$	0.133	0.042	0.045	93	−0.268	0.076	0.113	0.069	0.070	96	−0.238	0.110
		${\sigma }_{\gamma ,\gamma }^2$	0.088	0.021	0.027	91	0.682	0.005	0.109	0.034	0.033	90	0.694	0.006
		${\sigma }_{v,\gamma }^2$	−0.023	0.019	0.025	97	−0.336	0.016	−0.0	0.028	0.030	94	−0.338	0.026
		$\alpha$	0.000	0.131	0.136	94	−0.075	0.079	−0.003	0.144	0.176	96	−0.090	0.078
		$\omega$	−0.001	0.004	0.004	97	−0.081	0.006	−0.002	0.005	0.007	97	−0.066	0.007

(RB): relative bias; (SD): standard deviation; (SE): average bootstrap standard error; (CP): coverage probability of the 95% bootstrap confidence intervals; (Semiparametric): our proposed RL method; (Linear): RL model with $f\left(x\right)=cx$.

Table 2.

Summary of the estimated nonparametric reward sensitivity function in 200 simulations with Bernoulli (Case I) and Beta (Case II) reward distribution.

			Case I						Case II
			Semiparametric				Linear		Semiparametric				Linear
$\mathit{T}$	$\mathit{n}$		RB	SD	SE	CP	RB	SD	RB	SD	SE	CP	RB	SD
100	100	$f(-1.0)$	−0.021	0.188	0.221	98	−0.191	0.153	−0.024	0.312	0.308	99	−0.148	0.213
		$f(-0.5)$	−0.028	0.171	0.177	97	−0.341	0.077	−0.094	0.236	0.229	98	−0.308	0.107
		$f\left(0.5\right)$	−0.001	0.188	0.185	97	0.341	0.077	0.056	0.258	0.246	99	0.308	0.107
		$f\left(1.0\right)$	0.006	0.167	0.181	96	0.191	0.153	0.022	0.260	0.282	99	0.148	0.213
		$f\left(1.5\right)$	0.010	0.179	0.194	98	0.047	0.230	0.022	0.208	0.306	97	0.005	0.320
		$f\left(2.0\right)$	−0.025	0.277	0.284	97	−0.090	0.307	−0.030	0.293	0.427	98	−0.149	0.426
100	250	$f(-1.0)$	−0.013	0.110	0.131	99	−0.206	0.078	−0.016	0.179	0.190	97	−0.177	0.120
		$f(-0.5)$	−0.021	0.124	0.122	95	−0.353	0.039	−0.075	0.178	0.169	96	−0.331	0.060
		$f\left(0.5\right)$	0.002	0.136	0.136	93	0.353	0.039	0.067	0.197	0.179	95	0.331	0.060
		$f\left(1.0\right)$	−0.001	0.100	0.111	94	0.206	0.078	0.033	0.173	0.182	99	0.177	0.120
		$f\left(1.5\right)$	0.008	0.088	0.089	97	0.065	0.117	0.020	0.125	0.143	96	0.031	0.181
		$f\left(2.0\right)$	0.000	0.126	0.136	98	−0.068	0.156	−0.014	0.195	0.199	98	−0.109	0.241
500	100	$f(-1.0)$	0.007	0.083	0.082	98	−0.235	0.030	−0.013	0.112	0.123	96	−0.228	0.027
		$f(-0.5)$	0.018	0.040	0.047	97	−0.375	0.015	−0.008	0.066	0.082	99	−0.370	0.013
		$f\left(0.5\right)$	−0.023	0.047	0.054	98	0.375	0.015	0.002	0.080	0.097	99	0.370	0.013
		$f\left(1.0\right)$	−0.001	0.057	0.060	96	0.235	0.030	0.026	0.084	0.100	95	0.228	0.027
		$f\left(1.5\right)$	−0.006	0.048	0.051	96	0.100	0.044	0.001	0.055	0.060	98	0.091	0.040
		$f\left(2.0\right)$	−0.006	0.060	0.066	97	−0.028	0.059	−0.010	0.068	0.084	96	−0.039	0.053
500	250	$f(-1.0)$	−0.002	0.046	0.049	97	−0.237	0.018	−0.003	0.064	0.071	95	−0.226	0.018
		$f(-0.5)$	0.011	0.026	0.028	94	−0.377	0.009	0.008	0.034	0.040	95	−0.368	0.009
		$f\left(0.5\right)$	−0.010	0.029	0.032	94	0.377	0.009	−0.003	0.040	0.048	97	0.368	0.009
		$f\left(1.0\right)$	0.009	0.031	0.036	96	0.237	0.018	0.017	0.044	0.050	94	0.226	0.018
		$f\left(1.5\right)$	−0.004	0.028	0.031	94	0.102	0.027	−0.003	0.028	0.033	97	0.089	0.027
		$f\left(2.0\right)$	−0.007	0.036	0.038	95	−0.026	0.037	−0.011	0.041	0.047	95	−0.041	0.036

(RB): relative bias; (SD): standard deviation; (SE): average bootstrap standard error; (CP): coverage probability of the 95% bootstrap confidence intervals; (Semiparametric): our proposed RL method; (Linear): RL model with $f\left(x\right)=cx$.

Comparing semiparametric RL estimation results among four data size scenarios for both reward-generating mechanisms, we found that the estimation bias and standard deviation decrease significantly when we increase the number of trials from 100 to 500. The estimation bias and standard deviation also decrease when we increase the subject number from 100 to 250. The reduction rate is relatively large for $T=100$, compared to $T=500$. The sources of bias come from the bias of using 3 × 3 bivariate Gauss–Hermite quadrature to approximate (6) and the bias of using numerical optimizer to find the maximum of (7). By comparing reward-generating distribution in Cases I and II, we found that the distribution of the reward slightly affects the estimation performance. The simulation results show Case I has a smaller relative bias than Case II. In the Supplementary Materials Section S.1.5, we performed an additional simulation study involving three states (i.e., $m=3$). The results indicate that the Semiparametric RL model with three states also produces accurate estimation results.

4. Application to EMBARC Study

We now apply the proposed methodology to our motivating PRT data. Through a preliminary analysis, we found that the learning pattern might change from the first block to the second block. To avoid bias, we only focus on the PRT data in the first block (first 100 trials) in the paper.

First, we compare reward learning abilities between patients with MDD and the healthy control group. We fitted our proposed semiparametric reward learning models for the ‘MDD’ group that contains subjects diagnosed with MDD at the baseline (pre-treatment) and the ‘Control’ group that contains the healthy subjects with two repeated measurements (no treatment at both times). We examined the number of interior knots from 3 to 10, the boundary knots equal to {−4, 6}, and order $r=2$ for the I-Splines. By the AIC criterion, we selected the model with 6 interior knots. The parameters of interest are the transformed group learning rate ${\mu }_v$ and the group reward sensitivity function $f(\cdot )$. Table 3 shows the estimation results for ${\mu }_v$ and the elements in $\Sigma$. For inference, nonparametric bootstrap was applied to generate 200 resampling sets for each group. Bootstrap standard errors and 95% bootstrap confidence intervals were shown in Table 3. The results show that the learning rates for the two groups have similar values. We also constructed the 95% confidence interval for the contrast of ${\mu }_v$ between MDD and the control group; according to the result that the 95% confidence interval is equal to (−0.94, 1.93), we conclude that the difference of learning rate between MDD group and control group is not significant. Another interesting finding is that the learning rate is negatively correlated with reward sensitivity for the subjects.

Table 3.

Estimation of the parameters in EMBARC study under the proposed method for MDD and Control group.

	MDD			Control
Parameters	Est.	Err.	95% CI	Est.	Err.	95% CI
${\mu }_v$	−2.57	0.46	(−3.47, −1.67)	−3.07	0.67	(−4.38, −1.75)
${\sigma }_{v,v}^2$	10.68	3.75	(4.41, 25.86)	9.08	4.27	(3.06, 26.95)
${\sigma }_{\gamma ,\gamma }^2$	2.66	0.96	(1.50, 4.74)	3.15	2.30	(0.92, 10.86)
${\sigma }_{v,\gamma }^2$	−1.87	0.86	(−3.55, −0.19)	−2.92	0.99	(−4.85, −0.98)

Figure 3 presents the nonparametric estimation of group reward sensitivity function $f(\cdot )$. Figure 3(a) compares the nonparametric estimate of $f(\cdot )$ between MDD and Control group using the proposed method. The fitted reward sensitivity functions for both groups are clearly nonlinear. The function flattens when $x>2$ or $x<-2$, suggesting that subjects’ probability of correct actions at given states would not rise endlessly even if they received enough rewards. Meanwhile, the reward sensitivity functions for the two groups have similar shapes when $x<1.5,f\left(x\right)$ for the control group increases to a higher level than $f\left(x\right)$ for the MDD group when $x>2$. It suggests that the control group has a larger probability of taking correct actions at rich reward states than the MDD group when subjects in both groups receive adequate rewards in rich reward states. Figure 3(b) presents the 95% pointwise confidence band (PCB) and the 95% simultaneous confidence band (SCB) for the contrast of $f(\cdot )$ between the two groups. We evaluate $f\left(x\right)$ for $-3<x<4$ since few $Z_{it}$ values fall outside this interval. The construction of the PCB uses bootstrap and normal approximation, and the construction of the SCB uses a bootstrap method that mimics Hall and Horowitz (2013). For further details regarding the SCB and its coverage rates in simulation studies, see Supplementary Materials Section S.1.4. Because the 95% SCB covers zero in the entire range of $x$, we lack statistical evidence to conclude a significant difference in the reward sensitivity function between the Control and MDD groups. However, the 95% PCB suggests that the Control group may exhibit a higher reward sensitivity compared to the MDD group, particularly when the expected difference in reward between two stimuli is large (e.g., $x>2$). As an exploratory analysis, we constructed a 95% SCB for the contrast of $f(\cdot )$ between the two groups specifically at $x>2$. We observe that the entire band falls below zero for $x>2$. The result is shown in Figure S3 in the Supplementary Materials. This finding provides statistical evidence that the Control group exhibits a greater reward sensitivity compared to the MDD group, given that the expected difference in reward (rich reward minus lean reward) between two stimuli exceeds 2.

Fig. 3.

(a) The estimated reward sensitivity function estimates for MDD and Control group; (b) The estimation (red curve), 95% pointwise confidence band (shaded area in gray) and 95% simultaneous confidence band (two dash-dotted lines in blue) for the contrast of $f(\cdot )$ between the MDD and Control groups.

Moreover, we applied the proposed model with a linear reward sensitivity function represented by $f\left(x\right)=cx$ to both the MDD and Control groups. Figure S4 in the Supplementary Materials displays the estimated linear reward sensitivity functions. Compared to Figure 3 obtained from the semiparametric model, the linear model yields less precise reward sensitivity patterns and a higher AIC. Furthermore, the linear model fails to capture the substantial difference in the reward sensitivity function between the two groups beyond $x>2$.

The comparison of MDD versus Control examines whether reward sensitivity is a characteristic of MDD that differs between patients and controls. Our results suggest that reward sensitivity obtained from the probabilistic reward task may be considered a behavioral marker or a phenotype of depression. This analysis does not establish a causal relationship between depression and reduced reward sensitivity due to confounding.

Next, we compared reward learning abilities for the MDD patients between the sertraline (an antidepressant) (SERT) and placebo group (PBO). We fitted our proposed semiparametric reward learning models for subjects in PBO and SERT at week 0 and week 1, respectively. We used the same knots and order for the I-Splines as in the above analysis. Table 4 shows the estimated group learning rate and variance components in SERT and placebo group pre- and post-treatment. To investigate whether there is a significant difference in group learning rate ${\mu }_v$ for MDD patients who received SERT versus PBO, we constructed 95% bootstrap confidence interval of the difference between the change of ${\mu }_v$ from week 0 to week 1 in the two treatment arms. The result shows that the $95\%$ confidence interval is (−1.01, 3.10), indicating that the one-week changes in learning rate between PBO and SERT groups are not significantly different. A recent meta-analysis of the PRT test also reached the same conclusion of a non-significant group learning rate (Pike and Robinson, 2022).

Table 4.

Estimation of the parameters in EMBARC study for treatment (SERT) and placebo (PBO) group at pre- and post-treatment.

	SERT (pre-treatment)			SERT (post-treatment)
Parameters	Est.	Err.	95% CI	Est.	Err.	95% CI
${\mu }_v$	−1.84	0.77	(−3.35, −0.33)	−3.17	0.41	(−3.96, −2.37)
${\sigma }_{v,v}^2$	6.45	4.03	(1.60, 25.96)	8.90	4.11	(3.15, 25.12)
${\sigma }_{\gamma ,\gamma }^2$	2.92	1.40	(1.12, 7.65)	4.44	1.20	(1.99, 9.89)
${\sigma }_{v,\gamma }^2$	−2.14	1.31	(−4.70, 0.43)	−3.62	1.61	(−6.77, −0.46)
	PBO (pre-treatment)			PBO (post-treatment)
Parameters	Est.	Err.	95% CI	Est.	Err.	95% CI
${\mu }_v$	−3.03	0.42	(−3.85, −2.22)	−3.32	0.53	(−4.35, −2.28)
${\sigma }_{v,v}^2$	13.30	3.75	(6.32, 27.97)	5.24	2.22	(2.11, 13.04)
${\sigma }_{\gamma ,\gamma }^2$	4.16	1.50	(1.99, 8.70)	3.09	1.42	(1.42, 6.75)
${\sigma }_{v,\gamma }^2$	−2.46	0.99	(−4.41, −0.52)	−1.19	0.63	(−2.43, 0.06)

Figure 4(a) and 4(b) show the estimated reward sensitivity function $f(\cdot )$ for SERT and PBO groups at pre- and post-treatment. We find that the placebo group pre-(week 0) and post-treatment (week 1) reward sensitivity function is similar across the entire support, while for the SERT group, $f\left(x\right)$ increases to a higher level after treatment when $x>1.5$. It suggests that the SERT treatment increases MDD patients’ probability of taking correct actions at rich reward states when adequate rewards are received. We also constructed the 95% PCB and SCB of the difference between the change of $f(\cdot )$ pre- and post-treatment in the two groups. Figure 4(c) does not show sufficient statistical evidence that the pre-post change in reward sensitivity function differs between the two treatment groups. However, the presence of a similar pattern in the comparisons of reward sensitivity between Figure 3(b) and Figure 4(c) suggests that there might be a positive impact of sertraline on MDD patients, potentially bringing their reward learning sensitivity closer to that of healthy individuals at the rich state, which is worth future investigation. Regarding the timing of post-treatment measurement, the antidepressant is expected to take full effect in about four weeks. The rationale for measuring PRT reward sensitivity one-week post-treatment is to detect early responses. With a longer period of treatment, the difference may be greater.

Fig. 4.

(a) The estimated reward sensitivity function for SERT group pre- and post-treatment; (b) The estimated reward sensitivity function for PBO group; (c) The estimation (red curve), 95% PCB (shaded area in gray) and 95% SCB (two dash-dotted lines in blue) of the difference between PBO and SERT groups in the pre-post change of reward sensitivity function $f(\cdot )$ (difference in difference).

To investigate whether dysfunctions in the human brain circuitry are associated with decision-making and learning ability, we examined correlations between subjects’ learning rate, relative sensitivity, and task functional magnetic resonance imaging (fMRI) measures of brain activation in an emotional conflict task assessing amygdala-anterior cingulate (ACC) circuitry (Etkin et al., 2006). We find that both reward sensitivity and learning rate are associated with brain activities in negative affect circuitry evoked by sad stimuli. Detailed analysis can be found in Supplementary Materials Section S.2.1.

5. Discussion

In this paper, we propose a semiparametric inverse reinforcement learning framework to characterize reward-based decision-making with an application of probabilistic reward tasks in the EMBARC study. We assume that a patient’s decision-making process relies on two subject-specific factors, learning rate and reward sensitivity, and a shared non-decreasing function with a nonparametric form. Extensive simulation studies showed that our proposed method is satisfactory for large sample sizes under different reward-generating distributions. It also shows the advantage of the semiparametric structure when the true underlying function is beyond the restrictive parametric form using a sigmoid function. In the application, we find that MDD patients and control and patients between SERT and PBO groups have similar learning rates. However, the different groups have different reward sensitivity functions. The results in the paper are consistent with the findings in Huys et al. (2013), but by contrast, the proposed model shows a more detailed description of how the reward sensitivity differs. We also find that behavioral phenotypes, including learning rate and reward sensitivity, are associated with human brain activities at the negative affect circuits.

Estimation is carried out by maximizing the joint conditional log-likelihood for all patients at all trials, where the non-decreasing function and its first-order derivative are characterized by I-splines and M-splines. Because of the close relationship between M-splines and B-splines, M-splines and I-splines share good approximation power as B-splines. We studied asymptotic consistency and asymptotic distributions for the parameters in the proposed model. Note that we assume that $n$ (the number of patients) goes to infinity and $T$ (the number of trials) is fixed in the theoretical setting. It is much more challenging to assume both $n$ and $T$ go to infinity, and we will study this setting in future work. The proposed model can also be regarded as a time-varying model where the reward sensitivity is allowed to change through time. One future direction is allowing the learning rate to evolve gradually.

Our proposed work aligns with imitation learning or behavioral cloning in the RL literature. A related line of research is inverse RL (Ng et al., 2000; Abbeel and Ng, 2004; Arora and Doshi, 2021; Luckett et al., 2021), which assumes an agent’s behavior follows an optimal policy and seeks to learn the corresponding reward function from the agent’s observed behavior and decision-making process. In contrast, the problem we study here aims to recover the agent’s reward sensitivity function and learning rate without assuming the agent’s behavior follows an optimal policy.

Our current work does not accommodate heterogeneity in $Q_{it}$ beyond the aspects of learning rate and reward sensitivity. An extension is to model them in a mixed effects model framework but at the cost of more complex modeling and additional assumptions. Similarly, covariates can be introduced to model systematic heterogeneity in learning rates, reward sensitivity, and other parameters. Some recent work shows that the perceptual decision-making process would alternate between multiple interleaved strategies. Ashwood et al. (2022) provides an example in which the decision process is a mixture model of “engaged” and “lapse” trials. When the decision-making strategy is “engaged”, a subject chooses according to the RL model; When the decision-making strategy is “lapse”, the subject ignores the stimulus and chooses based on a fixed probability. The strategy-switching is characterized by a hidden Markov model. Extension to a semiparametric inverse reinforcement learning framework that allows strategies-switching is of interest. Another direction worth pursuing is a broader reinforcement learning context where states are not given randomly so that state distributions depend on past actions. Lastly, the current model is not restricted to characterizing decision-making in MDD patients. Our flexible framework can be easily applied to model other mental disorders such as schizophrenia and translates neuroscience and behavioral science constructs to clinical applications (Huys et al., 2016).

Appendix

A.1. Representation of M-splines and I-splines

According to Ramsay (1988), for a set of M-splines in $[L,U]$ with $K$ basis functions and order $r$, denote the $K$ basis as $M_k(\cdot ):=M_k(\cdot \mid r),k=1,\dots ,K$. Denote the knot sequence of the M-splines as $\left\{{\tau }_1,{\tau }_2,\dots ,{\tau }_{K+r}\right\}$, where the boundary knots should satisfy ${\tau }_1=\dots ={\tau }_r=L$ and ${\tau }_{K+1}=\dots ={\tau }_{K+r}=U$; the interior knots should satisfy $L<{\tau }_{r+1}<\dots <{\tau }_K<U$. (Please refer to Ramsay (1988) for a more general case that allows tie for the interior knots.) The M-splines can be constructed by recursion

$$\left\{\begin{array}{l}{M}_k(x\mid 1)=\frac{1}{{\tau }_{k+1}-{\tau }_k}I\left({\tau }_k\le x<{\tau }_{k+1}\right),\\ ⋮\\ M_k(x\mid r)=\frac{r\left[\left(x-{\tau }_k\right)M_k(x\mid r-1)+\left({\tau }_{k+r}-x\right)M_{k+1}(x\mid r-1)\right]}{(r-1)\left({\tau }_{k+r}-{\tau }_k\right)}.\end{array}\right.$$

It can be shown that $M_k(x\mid r)>0$ only when ${\tau }_k\le x<{\tau }_{k+r}$. Also $\int M_k(x\mid r)dx=1$ for all $k$. Define a set of I-splines as the integral of the corresponding set of M-splines, i.e.

$$I_k(x\mid r)={\int }_L^x{M}_k(u\mid r)du,\text{ for }k=1,\dots ,K,$$

$I_k(x\mid r)$ is guaranteed to be non-decreasing. A function represented by I-splines of order $r$ is $r-1$ times continuously differentiable, and the $r$-th derivatives are bounded in $[L,U]$; A function represented by M-splines of order $r$ is $r-2$ times continuously differentiable, and the $(r-1)$-th derivatives are bounded in $[L,U]$. Finally, a function represented by M-splines can also be represented by B-splines; to be more specific, a set of B-splines can be expressed by a set of M-splines with the same knots and order, i.e., $B_k(x\mid r)=\left({\tau }_{k+r}-{\tau }_k\right)M_k(x\mid r)/r$, hence M-splines and I-splines maintain good approximation power as B-splines (Schumaker, 2007). In practice, AIC can be used as the model selection criterion. A grid search on spline orders and interior knots can be conducted to select the I-splines with the smallest AIC value.

A.2. Asymptotic Theory

To simplify the notation, denote $\mathit{\theta }=\left(\mathit{\alpha },\mathbf{\Omega },{\mu }_v,\mathbf{\Sigma }\right)\in {\mathbb{R}}^d$. Let ${\mathit{\theta }}_0$ and $f_0(\cdot )$ denote the true parameters and nonparametric function. The log-likelihood function from a single subject is

$$\ell (\mathit{\theta },f)=\text{log}\left(\iint q(\mathit{a},\mathit{\theta },f)d\mathit{a}\right),$$ (10)

where $\mathit{a}=(v,\gamma )$

$$q(\mathit{a},\mathit{\theta },f)=\prod _{t=1}^T\frac{\text{exp}\left(A_{t}f\left(Z_t\right)\right)}{1+\text{exp}\left(f\left(Z_t\right)\right)}\varphi (\mathit{a};\mathit{\theta }).$$ (11)

For any vector $\mathit{b}\in {\mathbb{R}}^d$ and function $h\in {\mathbb{L}}_2[L,U]$ that has bounded $r$-th derivatives, we define the score operators of $\ell (\mathit{\theta },f)$ as $l_{\theta }^{\top }(\mathit{\theta },f)\mathit{b}$ and $l_f(\theta ,f)[h]$ by differentiating the log-likelihood function with respect to $\mathit{\theta }$ and $f$ along the submodel $\mathit{\theta }+ϵ\mathit{b}$ and $f+ϵh$. Define the information operator of $\ell (\mathit{\theta },f)$ as $ℐ(\mathit{\theta },f)=\text{E}\left\{{\left(l_{\mathit{\theta }},l_f\right)}^{*}\left(l_{\mathit{\theta }},l_f\right)\right\}$, where ${\left(l_{\mathit{\theta }},l_f\right)}^{*}$ is the dual operator of ${\left(l_{\mathit{\theta }},l_f\right)}^{*}ℐ(\mathit{\theta },f)$ is a self-adjoint operator from

$$ℰ=\left\{\left(\mathit{b},h\right):\mathit{b}\in {\mathbb{R}}^d,h\in {\mathbb{L}}_2\left[L,U\right]\text{ with bounded r-th derivatives}\}\right.$$

to itself. Define ${ℐ}^{1/2}(\mathit{\theta },f)$ as the square root operator of $ℐ(\mathit{\theta },f)$ such that $ℐ={ℐ}^{1/2}\circ {ℐ}^{1/2}$, where ${ }^{\circ }$ is the operator composition. To maximize (7), we only consider nonparametric function in

$${𝒮}_n=\left\{\tilde{f}:\tilde{f}(x)=\sum _{k=1}^{K_n}{\tilde{\eta }}_k\left\{I_k\left(x\right)-I_k\left(0\right)\right\},{\tilde{\eta }}_k\in \mathbb{R}\right\}.$$

The following conditions are needed for the theorems in this paper.

Condition 1. The true values ${\mathit{\theta }}_0$ lie in the interior of a known compact set in ${\mathbb{R}}^d$. The true parameter $f_0(\cdot )$ is increasing function with $f_0\left(0\right)=0$ in a known bounded interval $[L,U]$, where $L<0,U>0.f_0\left(x\right)\equiv f_0\left(U\right)$ for $x>U,f_0\left(x\right)\equiv f_0\left(L\right)$ for $x<L.f_0(\cdot )$ is $r-1$ times continuously differentiable and the $r$-th derivatives is bounded in $[L,U]$ where $r\ge 2$.

Condition 2. Let ${\tilde{f}}_0$ be in the projection of $f_0$ on ${𝒮}_n$ under ${ℒ}_2$ norm. We assume that there exists a constant $C>0$ such that ${\tilde{f}}_0$ belongs to a subspace of ${𝒮}_n$:

$${𝒮}_n^{*}\left(C\right)=\left\{\tilde{f}:\tilde{f}(x)=\sum _{k=1}^{K_n}{\tilde{\eta }}_k\left\{I_k\left(x\right)-I_k\left(0\right)\right\},0\le {\tilde{\eta }}_k<C{K}_n^{-1}\cdot \right\}.$$

Condition 3. For any two different set of parameters $\left({\mathit{\theta }}_1,f_1\right)$ and $\left({\mathit{\theta }}_2,f_2\right)$, the log-likelihood function $\ell \left({\mathit{\theta }}_1,f_1\right)\ne \ell \left({\mathit{\theta }}_2,f_2\right)$ with positive probability. The information operator $ℐ(\mathit{\theta },f)$ of the log-likelihood function $\ell (\mathit{\theta },f)$ at true parameter values $\left({\mathit{\theta }}_0,f_0\right)$ is invertible and satisfies

$$C_{\text{min}}\left(\parallel \mathit{b}{\parallel }_2^2+\parallel h{\parallel }_{{ℒ}_2}^2\right)\le \Vert {∥{ℐ}^{\frac{1}{2}}\left({\mathit{\theta }}_0,f_0\right)\left[\mathit{b},h\Vert \right]∥}_{{ℒ}_2\left(P\right)}^2\le C_{\text{max}}\left(\parallel \mathit{b}{\parallel }_2^2+\parallel h{\parallel }_{{ℒ}_2}^2\right),\forall (\mathit{b},h)\in ℰ,0<C_{\text{min}}<C_{\text{max}}<\infty .$$

Condition 4. The number of I-spline basis $K_n$ satisfies $K_n^{2r-2}{n}^{-1/2}\to \infty$ and $K_n^{1/2}{n}^{-1/2}\to 0$. The adjacent distance of the interior knots is between ${ }^{c^{-1}{K}_n^{-1}}$ and ${ }^{c{K}_n^{-1}}$ for some constant $c>1$.

Condition 1 ensures function $f_0$ is bounded and smooth. Condition 2 and 4 guarantees $\tilde{f}$ and ${\tilde{f}}^{\prime }$ to be uniformly bounded for all $\tilde{f}\in {𝒮}_n^{*}\left(C\right)$, where ${\tilde{f}}^{\prime }$ is the first order derivatives of $\tilde{f}$. The first part of Condition 3 is the identifiability, and the second part implies that the information operator is invertible in ${ℒ}_2$ space. In particular, this identifiability condition implies that the latent variable, $Z_{it}$ defined in equation (4), does not degenerate and should have a continuous density with support containing $[L,U]$, the domain of $f$. In Condition 4, we may set $K_n=n^{\delta }$, where $\frac{1}{4(r-1)}<\delta <1$.

We state the consistency and asymptotic distribution of the estimators for the model parameters in the following two theorems, whose proofs are given in the supplementary materials Section S.3.

Theorem 1. Under Conditions 1–4, ${∥\mathit{\theta }-{\mathit{\theta }}_0∥}_2\to 0,{∥\hat{f}-f_0∥ℒ}_2\to 0$ in probability. Furthermore, ${∥\mathit{\theta }-{\mathit{\theta }}_0∥}_2^2+{∥\hat{f}-f_0∥}_{{ℒ}_2}^2=o_p\left(n^{-1/2}\right)$.

To describe the asymptotic distributions of $\hat{f}$ and $\mathit{\theta }$, we first introduce the sets ${ℱ}_f=\left\{h\right(x):h(x\left)\text{has the r-th derivatives bounded by}1\text{in}\right[L,U\left]\right\}$. Define ${𝒪}_{\theta }$ be the unit ball in ${\mathbb{R}}^d$. We then treat $\left\{\mathit{\theta }-{\mathit{\theta }}_0,\hat{f}-f_0\right\}$ as a stochastic class in $l^{\infty }\left({𝒪}_{\theta }\times {ℱ}_f\right)$ whose value for $\left({\mathit{b}}^{*},h^{*}\right)$ is defined as

$${\mathit{b}}^{*\top }\left(\mathit{\theta }-{\mathit{\theta }}_0\right)+\int h^{*}\left(u\right)\left\{\hat{f}\left(u\right)-f_0\left(u\right)\right\}du,$$

where $\mathit{b}\in {𝒪}_{\theta },h\in {ℱ}_f,\left[{\mathit{b}}^{*},h^{*}\right]={ℐ}_{({\mathit{\theta }}_0,f_0)}[\mathit{b},h]$. The following theorem shows weak convergence of this stochastic process.

Theorem 2. Under Conditions 1–4, $n^{1/2}\left\{\mathit{\theta }-{\mathit{\theta }}_0,\hat{f}-f_0\right\}$ converges in distribution to a zero-mean and tight Gaussian process in the metric space $l^{\infty }\left({𝒪}_{\theta }\times {ℱ}_f\right)$ as $n\to \infty$.

The proof to Theorem 2 also implies the asymptotic normality of $\hat{\theta }$ and some smooth functional of $f$. However, it does not support pointwise normality, although the simulation study shows normality is still plausible. Also note that the convergence rate given in Theorem 1 is an intermediate result needed for proving the faster rate and asymptotic normality for $\mathit{\theta }$ in Theorem 2. On the other hand, the convergence rates for $\hat{f}$ in these two theorems are not exactly comparable. Theorem 1 gives the convergence rate for ${ℒ}_{2-}$ metric for $\hat{f}$; while the convergence rate and normality for $\hat{f}$ in Theorem 2 refer to the functional of $\hat{f},\int h^{*}\left(u\right)\hat{f}\left(u\right)du$.