Effects of stimulus salience on touchscreen serial reversal learning in a mouse model of fragile X syndrome

Abstract

Fragile X syndrome (FXS) is the most common inherited form of intellectual disability in males and the most common genetic cause of autism. Although executive dysfunction is consistently found in humans with FXS, evidence of executive dysfunction in Fmr1 KO mice, a mouse model of FXS, has been inconsistent. One possible explanation for this is that executive dysfunction in Fmr1 KO mice, similar to humans with FXS, is only evident when cognitive demands are high. Using touchscreen operant conditioning chambers, male Fmr1 KO mice and their male wildtype littermates were tested on the acquisition of a pairwise visual discrimination followed by four serial reversals of the response rule. We assessed reversal learning performance under two different conditions. In the first, the correct stimulus was salient and the incorrect stimulus was non-salient. In the second and more challenging condition, the incorrect stimulus was salient and the correct stimulus was non-salient; this increased cognitive load by introducing conflict between sensory-driven (i.e., bottom-up) and task-dependent (i.e., top-down) signals. Fmr1 KOs displayed two distinct impairments relative to wildtype littermates. First, Fmr1 KOs committed significantly more learning-type errors during the second reversal stage, but only under high cognitive load. Second, during the first reversal stage, Fmr1 KOs committed significantly more attempts to collect a reward during the timeout following an incorrect response. These findings indicate that Fmr1 KO mice display executive dysfunction that, in some cases, is only evident under high cognitive load.

1. Introduction

Fragile X syndrome (FXS) is an X-linked dominant disorder caused by a mutation in the fragile X mental retardation gene (FMR1) [1]. FXS is the most common inherited form of intellectual disability in males [2] and the most common genetic cause of autism spectrum disorder [3, 4]. Studies using molecular diagnostic techniques suggest that the prevalence of FXS is 1 in 4000 males and 1 in 6000 females [2, 5]. In addition to intellectual disability, patients with FXS display characteristic physical features and behavioral symptoms such as a long face and large ears, macroorchidism, hyperactivity, cognitive impairments, childhood seizures, and symptoms of autism such as repetitive behavior, decreased attention, and poor eye contact [6].

FXS is caused by an expansion of a CGG repeat in the 5′ untranslated portion of the FMR1 gene resulting in transcriptional silencing. Consequently, patients with FXS do not produce the fragile X mental retardation protein (FMRP), the protein for which the FMR1 gene codes. It is this lack of FMRP that is believed to cause the symptoms of FXS [7]. In addition to FXS, FMRP has also been shown to be reduced in psychiatric disorders in which cognitive deficits are present such as autism, schizophrenia, bipolar disorder, and major depressive disorder [8]. However, the mechanisms by which the reduction or elimination of FMRP results in these cognitive deficits are unknown.

The Fmr1 knockout (KO) mouse was developed as a model system to study the biological mechanisms underlying FXS [9]. In these mice, the murine analog of the human FMR1 gene (Fmr1) is inactivated and, like humans with FXS, Fmr1 KO mice lack FMRP. Consequently, Fmr1 KOs display many of the characteristic features and behaviors found in humans with FXS such as hyperactivity, seizures, macroorchidism, and cognitive deficits [9–11]. The similarities between humans with FXS and Fmr1 KO mice is unsurprising considering that the FMR1 gene is highly conserved among species [12]. In particular, the murine homolog Fmr1 shows 97% homology with the human FMR1 gene in amino acid sequence [13]. For these reasons, Fmr1 KO mice are an appropriate animal model to study the biological mechanisms of FXS as well as the potential impact of reduced levels of FMRP in other mental disorders.

With respect to cognitive function in FXS, one domain which has received substantial attention is executive function, an umbrella term for the group of higher cognitive skills which enable goal directed behavior [14]. Multiple studies indicate that individuals with FXS are impaired on tasks which directly measure executive functions including inhibition, working memory, cognitive flexibility, attentional set-shifting, and planning [reviewed in 15]. Cognitive flexibility, a type of executive function considered to be a core deficit in FXS [16], is the ability to adapt behavior to changing environmental demands [17]. Human and rodent studies indicate that cognitive flexibility is dependent on multiple brain regions including the prefrontal cortex (PFC), cerebellum, striatum, and thalamus [18–22]. In humans with FXS, morphology of brain regions involved in behavioral flexibility has been shown to be abnormal [23–25]. Additionally, decreased size of the cerebellar vermis has been shown to correlate with both the degree of cognitive dysfunction and FMRP in individuals with FXS [26].

In rodents, cognitive flexibility has most frequently been assessed using reversal learning tasks [for review see 27]. In these tasks, an animal first learns a response rule (e.g., always respond to the same stimulus in a pairwise visual discrimination to receive a food reward). Once the rule is learned to a high level, the response contingencies are reversed such that the previously-correct response becomes the incorrect response and vice versa. The main index of cognitive flexibility in reversal learning tasks is the degree to which the animal persists in responding according to the previously-correct response rule following the rule reversal (i.e., perseveration). In addition to providing an assessment of cognitive flexibility, reversal learning tasks also provide an index of the ability to learn and maintain a novel response rule and the ability to reacquire a previously-learned response rule. Tests of reversal learning performance using Fmr1 KO mice have provided mixed results. Specifically, although some studies have reported reversal learning deficits in Fmr1 KO mice relative to wildtypes [9, 28], others have reported mild or no reversal learning deficits [29, 30]. These inconsistences raise questions about the usefulness of Fmr1 KO mice as an animal model to study the biological mechanisms of executive function in FXS.

One possible explanation for the apparent inconsistencies in these studies is that executive function deficits in Fmr1 KO mice are only evident under high cognitive load. Specifically, while Fmr1 KO and wildtype mice may perform equally well under certain experimental conditions, performance of Fmr1 KO mice may be impaired relative to wildtypes as the task becomes more difficult. This hypothesis is supported by studies in humans with FXS showing that executive dysfunction increases with cognitive load [31–35]. A number of studies also indicate that humans with FXS, in comparison to controls, fail to recruit additional brain areas to compensate for increasing task difficulty [36–39]. Thus, the human literature suggests that inconsistent findings of executive dysfunction in Fmr1 KO mice may be due to a failure to consider the effects of cognitive load. To our knowledge, the effects of cognitive load on executive function in Fmr1 KO mice have never been assessed.

In the present study, we used a serial reversal learning task performed in a touchscreen operant conditioning chamber to test the hypotheses that executive function in Fmr1 KO mice is impaired under conditions of high but not low cognitive load. Male Fmr1 KO mice (FVB.129P2-Pde6b⁺ Tyr^c-ch Fmr1^tm1Cgr/J) and their male wildtype littermates were tested on the acquisition of a pairwise visual discrimination followed by four serial reversals. We chose to use a serial reversal learning paradigm instead of a task composed of a single reversal because the use of multiple reversals allows for the assessment of additional cognitive abilities. For example, during the second reversal stage mice must reacquire the response rule that they first learned during the acquisition stage. Additionally, the use of multiple reversals allows for an assessment of the persistence of an observed deficit.

In order to manipulate cognitive load, we used stimuli of differential salience. Specifically, mice were tested under conditions in which (1) the correct stimulus was salient and the incorrect stimulus was non-salient or (2) the incorrect stimulus was salient and the correct stimulus was non-salient. In the second of these two conditions, sensory-driven (i.e., bottom-up) and task-dependent (i.e., top-down) signals conflicted, increasing cognitive load. We quantified stimulus salience by determining innate stimulus preference during the first session of the acquisition stage. In comparison to maze-based tasks, the use of touchscreen operant conditioning chambers to measure executive function facilitates cross-species comparison because virtually identical versions of touchscreen based reversal learning tasks have been developed for mice [40–42], rats [43–45], humans [46, 47], and non-human primates [48, 49].

2. Materials and Methods

2.1. Subjects

Experimental subjects were bred and maintained in the Department of Psychology at the University of Memphis. All experiments were approved by a local Institutional Animal Care and Use Committee and conducted in accordance with the National Institutes of Health Guidelines for the Care and Use of Laboratory Animals. Mice used to generate experimental subjects were originally purchased from the Jackson Laboratory (Bar Harbor, ME). Two phases of breeding were required to produce experimental mice. In the first phase, male mice which were hemizygous for the Fmr1^tm1Cgr targeted mutation (FVB.129P2-Pde6b⁺ Tyr^c-ch Fmr1^tm1Cgr/J; JAX mice stock number 004624) were mated with female wildtype mice on the same genetic background (FVB.129P2-Pde6b⁺ Tyr^c-ch/AntJ; JAX mice stock number 004828). This breeding protocol produced litters composed exclusively of heterozygous females and wildtype males. In the second phase, heterozygous female mice were mated with wildtype male mice to produce litters containing both hemizygous mutant and wildtype males. These hemizygous male mice and their male wildtype littermates were used as experimental subjects. Importantly, all mice used in the present study were homozygous for the 129P2/OlaHsd wildtype Pde6b allele and did not suffer from blindness due to retinal degeneration [50]. All breeding pens contained a single male and single female to enable correct identification of littermates. Genotyping of all mice used in the study was performed by Transnetyx (Cordova, TN). Mice were continuously maintained in a temperature controlled environment (21±1°C) on a 12:12 light:dark cycle (lights on at 0800) and were given free access to food and water until the beginning of the experiment, at which point mice were food restricted as detailed below.

2.2. Apparatus

Training and testing were conducted in four Med Associates (St. Albans, VT) mouse operant conditioning chambers (ENV-307W) enclosed in sound attenuating cubicles (ENV-022MD). Centrally mounted on the rear wall of each chamber was a liquid dipper (ENV-302W) which, when actuated, provided access to 0.01 cc of Silk Vanilla Soymilk (WhiteWave Foods; Broomfield, CO) through a 2.2 cm round opening. An infrared (IR) detector in the liquid dipper receptacle provided a means to detect head entries. A stimulus light (ENV-321W) was mounted above the liquid dipper receptacle, and a house light (ENV-315W) with bulb (CM1829; Chicago Miniature Lighting, LLC) was mounted above the stimulus light at the top of the chamber. Med Associates chambers were modified in-house by adding to the front wall of each chamber an IR touchscreen (NEX121) manufactured by Nexio (Incheon, South Korea) and provided by Lafayette Instruments (Lafayette, IN). As shown in Figure 1A, visual stimuli were presented on the screen and mice responded to these stimuli by nosepoking. Because nosepokes to the touchscreen were detected by a matrix of IR sensors, mice needed only to break the IR beam and did not have to apply pressure to the screen. Operant chambers were controlled by a Lafayette Instruments control unit running ABET II and Whisker software. All operant schedules were written in-house using ABET II, and visual stimuli were created using Adobe Photoshop.

Figure 1.

(A) One of the touchscreen operant conditioning chambers and the visual stimuli used in the present study. (B) During the first session of the acquisition stage, mice tested on the discrimination in which + was the correct stimulus (stimulus group 1: wildtype n = 11, Fmr1 KO n = 10) performed significantly worse (p < .001) than mice tested on the discrimination in which x was the correct stimulus (stimulus group 2: wildtype n = 9, Fmr1 KO n = 9). This indicates that the x stimulus was significantly more salient than the + stimulus.

2.3. Pairwise Visual Discrimination and Serial Reversal Learning

2.3.1. General procedures

Mice began the experiment at 12 weeks of age and were trained and tested in the same chamber and at the same time daily seven days a week until they completed the experiment. During training and testing, mice were maintained at 90% of baseline weight which was taken at the beginning of the experiment.

2.3.2. Training

Mice completed three training stages before testing began. Training stages lasted for 60 minutes or until mice reached criterion, whichever occurred first. In the first stage, the house light was always illuminated, the stimulus light was never illuminated, and stimuli were never shown on the touchscreen. The dipper arm was alternately raised for 20 seconds and then lowered for 20 seconds. The mouse reached criterion on this stage when he collected at least 20 rewards during a session.

In the second stage, a white square (6.5 cm x 6.5 cm) was randomly presented on the right or left of the screen. Following a nosepoke to the square, it disappeared from the screen and the dipper arm was raised for 10 seconds. The next trial began immediately following this. The mouse reached criterion on this stage when he collected at least 20 rewards in 60 minutes.

In the third and final training stage, a 5 second intertrial interval (ITI) and trial initiation requirement were added. The ITI, during which only the house light was illuminated, began immediately following the 10 second reward and lasted for 5 seconds. Following the ITI, the stimulus light was illuminated and the mouse was required to nosepoke the food receptacle to initiate the next trial. When a nosepoke occurred, the stimulus light was turned off and the white square appeared on the screen. The mouse reached criterion on the final training stage when he completed 80 trials in a single session.

2.3.3. Acquisition of the visual discrimination

The acquisition stage was identical to the third training stage with the following exception. Immediately after a trial was initiated by the mouse, two visual stimuli, each composed of two lines 6.5 cm long and 0.2 cm wide, were randomly presented on the right or left side of the screen. As shown in Figure 1A, one of these stimuli resembled an x and the other a +. The x stimulus was created by rotating the + stimulus 45 degrees. Thus, the two stimuli were identical in all respects (e.g., size, brightness) with the exception of orientation. During the acquisition stage, the x stimulus was the correct stimulus for half of the mice, and the + stimulus was the correct stimulus for the other half. A nosepoke to the correct stimulus resulted in access to the reward for 10 seconds. A nosepoke to the incorrect stimulus resulted in a 10 second timeout which was signaled by extinguishing the house light. Immediately following a nosepoke to either stimulus, the visual stimuli were removed from the screen. A 5 second ITI followed reward or timeout, at which point the trial initiation light above the food receptacle was illuminated signaling that the mouse could initiate another trial.

During each session, the second and all subsequent trials were considered either “correction” or “non-correction” trials depending on the correctness of the previous trial. Specifically, a correction trial followed an incorrect trial and a non-correction trial followed a correct trial. During correction trials, stimulus presentation was not randomized. Rather, the correct and incorrect stimuli were presented on the same side as in the previous trial. The purpose of this was to prevent the mouse from developing a strategy in which he ignored the visual stimuli, always chose the same side, and was therefore rewarded on 50% of the trials. A non-correction trial followed a correct trial, and stimulus presentation was randomized.

Sessions continued in this manner until 80 trials were completed or 60 minutes had elapsed, whichever occurred first. Both correction and non-correction trials were counted towards the 80 trial maximum per session. Mice reached criterion when they completed a single session at 80% correct (calculated using non-correction trials).

2.3.4. Serial reversals of the visual discrimination

Once mice reached criterion on the acquisition stage, they were tested on a series of four serial reversals. Serial reversal stages were identical to the acquisition stage with the exception that the response contingencies were reversed relative to the previous stage. Specifically, mice that were rewarded for nosepoking the + stimulus during the acquisition stage (stimulus group 1: wildtype n = 11, Fmr1 KO n = 10) were rewarded for nosepoking the x stimulus during reversal 1, the + stimulus during reversal 2, the x stimulus during reversal 3, and the + stimulus during reversal 4. Conversely, mice that were rewarded for nosepoking the x stimulus during the acquisition stage (stimulus group 2: wildtype n = 9, Fmr1 KO n = 9) were rewarded for nosepoking the + stimulus during reversal 1, the x stimulus during reversal 2, the + stimulus during reversal 3, and the x stimulus during reversal 4.

2.4. Dependent Variables

The following dependent variables were collected during the acquisition stage and the four reversal stages: [1] errors to criterion (excluding correction trials), [2] percent of correct trials during which the mouse collected a reward, [3] percent of incorrect trials during which the mouse attempted to collect reward, [4] latency to nosepoke the correct visual stimulus, [5] latency to nosepoke the incorrect visual stimulus, [6] latency to retrieve a reward following nosepoke to the correct visual stimulus, and [7] latency to attempt to retrieve a reward following a nosepoke to the incorrect visual stimulus. Attempts to collect a reward (indicated by a head entry into the food receptacle) during the 10 second timeout following an incorrect stimulus choice are referred to as reward collection errors. These errors indicate that the mouse attempted to collect a reward despite visual and auditory stimuli (i.e., absence of house light and dipper-arm motor) indicating that a reward was not available. Latency to nosepoke the visual stimulus was defined as the time (ms) between trial initiation (i.e., a nosepoke to the food receptacle that caused the visual stimuli to appear on the screen) and a nosepoke to either the correct or incorrect stimulus. Latency to collect or attempt to collect a reward was defined as the time (ms) between a nosepoke to the stimulus on the screen and the first head entry into the food receptacle during the 10 second reward or timeout.

During the reversal stages, all errors during a session were defined as perseverative or learning errors depending on performance during that stage [18, 43, 44]. This was done to differentiate perseveration to the previously-correct response rule from (1) learning a new response rule during the first reversal or (2) reacquiring a previously-learned response rule during the second, third, and fourth reversals. Specifically, errors committed during sessions in which performance was below chance levels (≤ 40% correct) were classified as perseverative errors, and errors committed during sessions in which performance did not differ from or was above chance (41% – 80% correct) were classified as learning errors.

2.4. Subject Attrition and Data Analysis

Two Fmr1 KOs and one wildtype died before completing the task and were dropped from all analyses. The remaining sample was composed of 19 Fmr1 KOs and 20 wildtypes. When considering the effect of stimulus salience, these groups were divided into stimulus group 1 [Fmr1 KOs (n = 10) and wildtypes (n = 11)] and stimulus group 2 [Fmr1 KOs (n = 9) and wildtypes (n = 9)]. The overall number of Fmr1 KOs and wildtypes in the experiment and the number of Fmr1 KOs and wildtypes in the two stimulus groups remained constant for all statistical tests presented in the manuscript.

All ANOVAs were conducted using the GLM or UNIANOVA command in SPSS. If the ANOVA contained a within subjects factor, the F statistic, the p value, the factor or interaction degrees of freedom, and the error degrees of freedom were reported from the Wilks’ Lambda row of the multivariate tests table. If an ANOVA contained only between subjects factors, these values were reported from the tests of between-subjects effects table. The criterion for statistical significance was p < .05. Fisher's Least Significant Difference procedure was used for all post hoc tests.

3. Results

3.1. Stimulus Salience

Using methods similar to the measurement of stimulus bias in a previous study using rats [51], we quantified stimulus salience by assessing the relative preference for each stimulus on the first session of the acquisition stage. We performed a 2 × 2 between-subjects analysis of variance (ANOVA) using percent correct as the dependent factor, rewarded stimulus (+ or ×) as the first between-subjects factor, and genotype (Fmr1 KO or wildtype) as the second between-subjects factor. ANOVA revealed a significant main effect of rewarded stimulus [F (1, 35) = 18.29, p < .001], but no significant main effect of genotype or significant interaction between rewarded stimulus and genotype. As shown in Figure 1B, post hoc comparisons indicated that mice tested on the discrimination in which x was the correct stimulus (wildtypes: M = 60.22%, SD = 6.35%; Fmr1 KOs: M = 58.04%, SD = 15.56%) performed significantly better (p < .001) than mice tested on the discrimination in which + was the correct stimulus (wildtypes: M = 45.24%, SD = 8.66%; Fmr1 KOs: M = 45.00%, SD = 8.32%). This indicates that the x stimulus was significantly more salient than the + stimulus. Thus, Fmr1 KOs (n = 10) and wildtypes (n = 11) that were rewarded for responding to the non-salient + stimulus during the acquisition stage (stimulus group 1) were required to stop responding to the salient x stimulus during the second and fourth reversal (difficult reversals), while Fmr1 KOs (n = 9) and wildtypes (n = 9) that were rewarded for responding to the salient x stimulus during the acquisition stage (stimulus group 2) were required to stop responding to that stimulus during the first and third reversal. Consequently, at each reversal stage, half of the mice were challenged with a relatively difficult reversal (i.e., non-salient correct stimulus, salient incorrect stimulus), and the other half were challenged with a relatively easy reversal (i.e., salient correct stimulus, non-salient incorrect stimulus). Because of this, we were able to compare the effect of the Fmr1 mutation on performance at each reversal stage under conditions of high cognitive load (difficult reversals) and low cognitive load (easy reversals). Importantly, on stages during which mice were required to select the salient stimulus, they committed significantly (p < .05) fewer errors to criterion (M = 133.46; SD = 69.89) than on stages during which they were required to select the non-salient stimulus (M = 165.17; SD = 95.66). This is consistent with the stimulus preference observed during the first session of the acquisition stage and the hypothesis that discriminations on which mice were required to select the non-salient stimulus and withhold responding to the salient stimulus were more difficult than discriminations on which mice had to learn to select the salient stimulus and withhold responding to the n-on-salient stimulus.

3.2. Stimulus Choice

To examine the effects of the Fmr1 mutation and stimulus salience on pairwise visual discrimination and serial reversal learning, we performed a 2 × 2 × 5 mixed-model ANOVA using errors to criterion as the dependent factor, genotype (wildtype or Fmr1 KO) and stimulus group (1 or 2) as between-subjects factors, and stage (acquisition - reversal 4) as a within-subjects factor. Repeated measures ANOVA revealed a significant genotype x stimulus group interaction [F (1, 35) = 4.15, p < .05], a significant stage x stimulus group interaction [F (4, 32) = 4.17, p < .01], and a significant main effect of stage [F (4, 32) = 6.06, p < .01].

To determine the cause of the significant interactions, we examined errors to criterion of Fmr1 KOs and wildtypes on the acquisition stage and on each of the four reversal stages. Post hoc tests indicated that regardless of stimulus salience, performance of Fmr1 KOs did not differ significantly from wildtype mice during the acquisition stage. However, as shown in Figure 2 panels A and B, Fmr1 KOs committed significantly more errors than wildtypes (p < .05) during the second reversal stage, but only when the reversal was of the difficult type (i.e., non-salient correct stimulus, salient incorrect stimulus). Performance of Fmr1 KOs and wildtypes did not differ significantly during any other reversal stage, irrespective of the salience of the correct stimulus.

Figure 2.

Performance of Fmr1 KO mice and wildtype littermates on the acquisition and four serial reversals of a pairwise visual discrimination. For stimulus group 1 (wildtype n = 11, Fmr1 KO n = 10), the correct stimulus was salient on reversals 1 and 3 (an easy reversal) and non-salient on reversals 2 and 4 (a difficult reversal). For stimulus group 2 (wildtype n = 9, Fmr1 KO n = 9), this pattern was reversed. (A and B) Fmr1 KO mice committed significantly more errors than wildtype littermates (* = p < .05) during the second reversal stage, but only when the correct stimulus was non-salient. (C) An analysis of the type of errors committed on reversal 2 by stimulus group 1 revealed that Fmr1 KO mice in stimulus group 1 did not commit significantly more perseverative errors (session performance ≤ 40% correct) than wildtype littermates, but did commit significantly more (* = p < .01) learning errors (session performance 41% – 80% correct).

As with a previous study using the Lurcher mouse [18], at each stage errors to criterion was strongly correlated with trials to criterion and sessions to criterion (correlation coefficients ranged between .97 and .99, p < .0000001 for all tests). With regard to observed differences between groups, for all three variables (1) Fmr1 KOs committed significantly more errors than wildtypes during the second reversal stage under the difficult condition and (2) performance of Fmr1 KOs and wildtypes did not differ significantly at any other stage, regardless of salience of the correct stimulus. The number of sessions required to complete the experiment is show below separately for each subgroup. Fmr1 wildtypes (n = 11) in stimulus group 1: [Acq: M = 9.82, SD = 4.55; Rev1: M = 9.73, SD = 5.10; Rev2: M = 7.67, SD = 1.73; Rev3: M = 6.67, SD = 2.19; Rev4: M = 7.33, SD = 2.04]; Fmr1 KOs (n = 10) in stimulus group 1: [Acq: M = 11.90, SD = 5.48; Rev1: M = 10.60, SD = 3.71; Rev2: M = 11.78, SD = 6.08; Rev3: M = 8.50, SD = 3.77; Rev4: M = 8.00, SD = 2.62]; Fmr1 wildtypes (n = 9) in stimulus group 2: [Acq: M = 9.89, SD = 3.98; Rev1: M = 16.89, SD = 5.66; Rev2: M = 11.00, SD = 5.29; Rev3: M = 12.00, SD = 9.11; Rev4: M = 9.11, SD = 6.93]; Fmr1 KOs (n = 9) in stimulus group 2: [Acq: M = 7.11, SD = 1.45; Rev1: M = 13.00, SD = 4.52; Rev2: M = 8.29, SD = 3.66; Rev3: M = 10.71, SD = 6.99; Rev4: M = 6.71, SD = 1.85].

To further probe the nature of the differential performance of Fmr1 KOs (n = 10) and wildtypes (n = 11) on the difficult reversal during the second reversal stage (stimulus group 1), we examined errors separately on sessions during which mice were still responding according to the response rule of the prior stage (i.e., overall session performance ≤ 40% correct) and on sessions during which mice were no longer responding according to the response rule of the prior stage (i.e., overall session performance between 41% and 80% correct). These errors were termed perseverative errors and learning errors, respectively. Specifically, we performed a 2 × 2 mixed-model ANOVA using genotype as a between-subjects factor, error type (perseverative and learning) as a within-subjects factor, and errors as the dependent factor. ANOVA indicated a significant interaction between genotype and error type [F (1, 19) = 10.40, p < .01], a main effect of genotype [F (1, 19) = 7.04, p < .05], and a main effect of error type [F (1, 19) = 36.29, p < .000001]. As shown in Figure 2C, post hoc comparisons indicated that the number of learning errors committed by Fmr1 KOs (M = 158.22, SD = 91.41) was significantly greater than those committed by wildtypes (M = 72.88, SD = 21.55). However, the number of perseverative errors committed by Fmr1 KOs (M = 35.22, SD = 21.22) and wildtypes (M = 35.67, SD = 14.86) did not differ significantly. This indicates that although stimulus perseveration of Fmr1 KOs and wildtypes did not differ on the second reversal stage, it was significantly more difficult for Fmr1 KOs to relearn a response rule (i.e., mice had previously learned this rule to criterion during the acquisition stage), but only under high cognitive load (i.e., non-salient correct stimulus, salient incorrect stimulus).

3.3. Latency: Stimulus Choice

We performed a 2 × 2 × 2 × 5 mixed-model ANOVA to examine the effects of genotype, stimulus salience, correctness, and stage on the latency to nosepoke the correct or incorrect stimulus following stimulus onset. We used latency to nosepoke the stimulus as the dependent factor, strain and stimulus group as between-subjects factors, and correctness (i.e., whether the mouse selected the correct or incorrect stimulus on the touchscreen) and stage as within-subjects factors. Repeated measures ANOVA revealed a significant main effect of stage [F (4, 32) = 12.90, p < .00001] and a significant main effect of correctness [F (1, 35) = 9.30, p < .01].

Post hoc comparisons revealed that latencies (s) to nosepoke the correct stimulus (Acq: M = 2.61, SD = 0.64; Rev1: M = 2.36, SD = 0.74; Rev2: M = 2.18, SD = 0.86; Rev3: M = 1.96, SD = 0.42; Rev4: M = 1.93, SD = 0.46) and incorrect stimulus (Acq: M = 2.61, SD = 0.71; Rev1: M = 2.28, SD = 0.65; Rev2: M = 2.10, SD = 0.72; Rev3: M = 1.87, SD = 0.36; Rev4: M = 1.81, SD = 0.37) did not differ on the acquisition stage or the first two reversal stages. However, latencies to nosepoke the correct stimulus were significantly longer than latencies to nosepoke the incorrect stimulus on the third (p < .01) and fourth (p < .05) reversal stage.

3.4. Reward Collection and Reward Collection Errors

We performed a 2 × 2 × 2 × 5 mixed-model ANOVA to examine the effects of genotype, stimulus-salience, correctness, and stage on the propensity of mice to attempt to collect a reward during the 10 second reward or timeout period following a nosepoke to the correct or incorrect stimulus. We used genotype and stimulus group as between-subjects factors and correctness and stage as within-subjects factors. The dependent variable was the percentage of trials during which the mouse broke the IR beam in the food receptacle during the 10 second reward or timeout period following a nosepoke to the correct or incorrect stimulus. The ANOVA revealed a significant correctness x stage x genotype interaction [F (4, 32) = 4.17, p < .05], a significant correctness x stage interaction [F (4, 32) = 12.76, p < .00001], a significant main effect of stage [F (4, 32) = 13.42, p < .00001], and a significant main effect of correctness [F (1, 35) = 1328.30, p < .00001]. There was no significant main effect of stimulus group or interaction between stimulus group and any other factor, indicating that stimulus salience did not affect the propensity to collect or attempt to collect a reward.

As shown in Figure 3A, post hoc tests revealed that the percentage of incorrect trials during which Fmr1 KOs committed a reward collection error on the acquisition stage (M = 46.41%, SD = 18.74%) was similar to that of wildtypes (M = 44.20%, SD = 15.23%). However, on the first reversal stage, the percentage of incorrect trials during which Fmr1 KOs committed a reward collection error (M = 36.98%, SD = 14.49%) was significantly greater than that of wildtypes (M = 26.58%, SD = 10.22%). To assess the nature of this differential performance, we examined reward collection errors across all stages in each genotype separately. Over the course of the experiment (Acquisition – Reversal 4), the percentage of trials during which mice committed a reward collection error declined to the same degree in both Fmr1 KOs (M = 19.43%, SD = 20.69%) and wildtypes (M = 19.39%, SD = 14.22%). However, although this decline occurred rapidly in wildtypes, it was significantly protracted in Fmr1 KOs. Specifically, as shown in Figure 3B, the percentage of trials during which wildtypes committed a reward collection error was similar, and did not differ significantly, during the first (M = 26.58%, SD = 2.29%) and fourth (M = 24.81%, SD = 3.19%) reversal stage. However, the percentage of trials during which Fmr1 KOs committed a reward collection error was substantially and significantly greater during the first reversal (M = 36.98%, SD = 14.50%) than it was during the fourth reversal (M = 26.98%, SD = 13.02%).

Figure 3.

Percentage of trials on which Fmr1 KO (n = 19) and wildtype littermates (n = 20) committed a food receptacle head entry following a nosepoke to the correct (rewarded) or incorrect (non-rewarded) stimulus displayed on the touchscreen. (A) During reversal 1, the percentage of trials during which Fmr1 KOs attempted to collect a reward following a nosepoke to the incorrect, non-rewarded stimulus (i.e., reward collection errors) was significantly greater than that of wildtypes (* = p < .05). (B) Although reward collection errors declined to the same degree across the experiment in both Fmr1 KOs and wildtypes, the decline was significantly protracted in Fmr1 KOs (* = p < .05).

3.5. Latency: Reward Collection and Reward Collection Errors

We performed a 2 × 2 × 2 × 5 mixed-model ANOVA to examine the effects of genotype, stimulus-salience, correctness, and stage on the latency of mice to collect or attempt to collect a reward during the 10 second reward or timeout period following a nosepoke to the correct or incorrect stimulus. We used genotype and stimulus group as between-subjects factors and correctness and stage as within-subjects factors. The dependent variable was latency (s) to break the IR beam in the food receptacle during the 10 second reward or timeout period. Repeated measures ANOVA revealed a significant correctness x stage interaction [F (4, 32) = 17.14, p < .000001], a significant main effect of stage [F (4, 32) = 9.55, p < .0001], and a significant main effect of correctness [F (1, 35) = 762.46, p < .00001].

Post hoc comparisons revealed two patterns in the reward collection latency data. First, at all stages, latency (s) to collect a reward following a nosepoke to the correct stimulus (Acq: M = 1.36, SD = 0.16; Rev1: M = 1.33, SD = 0.17; Rev2: M = 1.28, SD = 0.15; Rev3: M = 1.22, SD = 0.16; Rev4: M = 1.19, SD = 0.13) was significantly shorter than the latency to commit a reward collection error during the timeout following a nosepoke to the incorrect stimulus (Acq: M = 3.44, SD = 0.52; Rev1: M = 3.86, SD = 0.65; Rev2: M = 3.97, SD = 0.76; Rev3: M = 4.27, SD = 0.86; Rev4: M = 4.16, SD = 0.77). Second, latency to collect a reward following a nosepoke to the correct stimulus decreased significantly across stages, but latency to commit a reward collection error during the timeout following a nosepoke to the incorrect stimulus increased significantly across stages.

4. Discussion

In the present study, we assessed executive function in male Fmr1 KO mice (FVB.129P2-Pde6b⁺ Tyr^c-ch Fmr1^tm1Cgr/J) and their male wildtype littermates using a touchscreen-based serial reversal learning paradigm. After learning to nosepoke a visual stimulus presented on a touchscreen for a food reward, mice acquired a pairwise visual discrimination and subsequently completed four serial reversals of the response rule. In order to test the hypothesis that executive function in Fmr1 KOs is impaired only under conditions of high cognitive load, we assessed reversal learning performance under two different conditions. In the first and less difficult condition, the correct stimulus was salient. In the second and more challenging condition, the incorrect stimulus was salient, increasing the difficulty of the task. Specifically, when the incorrect stimulus was salient and the correct stimulus was non-salient, sensory-driven (i.e., bottom-up) and task-dependent (top-down) signals conflicted, increasing cognitive load. That is, top-down signals drove the mouse to respond to the correct stimulus, while bottom-up signals drove the mouse to respond to the incorrect stimulus. The increased conflict, and the necessity of resolving this conflict on a trial-by-trial basis, increased the difficulty of difficult reversals. During easy reversals, there was no conflict between top-down and bottom-up signals because task demands required the mouse to choose the stimulus to which he was naturally inclined to respond (i.e., the salient stimulus) and avoid the stimulus to which he was naturally disinclined to respond (i.e., the non-salient stimulus). Fmr1 KOs displayed two distinct impairments relative to wildtype littermates. First, Fmr1 KOs committed significantly more errors during the second reversal stage, but only on the more challenging condition (i.e., when cognitive load was high). An analysis of error type revealed that the differential performance of Fmr1 KOs and wildtypes occurred during late, but not early sessions. Second, during the first reversal stage, Fmr1 KOs committed significantly more attempts to collect a reward during the timeout following an incorrect response. This phenomenon was not related to stimulus salience.

4.1. Stimulus Choice Errors

During the acquisition stage, overall performance of Fmr1 KOs and wildtypes did not differ significantly. This was true regardless of the salience of the stimulus to which mice were required to respond. During the serial reversal stages, Fmr1 KOs committed significantly more errors than wildtypes on the second reversal stage, but only when the reversal was difficult (i.e., non-salient correct stimulus, salient incorrect stimulus). Subsequent analysis of these errors indicated that Fmr1 KOs and wildtypes did not differ in their degree of stimulus perseveration (see Figure 2). Rather, Fmr1 KOs committed significantly more learning-type errors once overall session performance was greater than or equal to chance. Performance of Fmr1 KOs and wildtypes did not differ significantly on any other reversal, regardless of stimulus salience.

Two issues should be noted with regard to the deficit in stimulus choice observed in Fmr1 KOs. First, although Fmr1 KOs were impaired during the second reversal stage on the difficult reversal, performance of Fmr1 KOs in both the easy and difficult reversal conditions was unimpaired on the first reversal stage. One explanation for this is that the observed deficit in Fmr1 KOs reflects an impaired ability to reacquire a previously-learned response rule (i.e., the response rule that they learned during the acquisition stage). Because reacquiring a previously-learned response rule occurs for the first time during the second reversal stage, an impairment in this ability would not be expected to result in impaired performance during the first reversal stage, regardless of the salience of the correct stimulus. That performance of Fmr1 KOs and wildtypes did not differ during the third and fourth reversal stage regardless of salience of the correct stimulus indicates that the observed impairment in Fmr1 KOs was relatively transient and therefore less severe.

A second issue to note is that the deficit in Fmr1 KOs was only evident during difficult reversals (i.e., when the non-salient stimulus was correct). One possible explanation for this is that the impaired ability of Fmr1 KOs to reacquire a previously-learned response rule only occurred under high cognitive load (i.e., when sensory-driven and task-dependent information conflicted). That this deficit only occurred under high cognitive load may explain why previous studies in Fmr1 KOs have sometimes failed to find reversal learning impairments in these mice. Additionally, this explanation is consistent with evidence indicating that humans with FXS show greater executive dysfunction under high cognitive load [15, 31–35, 52] and that humans with FXS fail to recruit additional brain regions to compensate for increasing task difficulty [36–39, 53–55].

It is important to point out, however, that the salience of the correct stimulus during the acquisition stage and first reversal stage differed in stimulus group 1 and stimulus group 2. Specifically, during the acquisition stage, the correct stimulus was non-salient for stimulus group 1 and salient for stimulus group 2, and during the first reversal stage, the correct stimulus was salient for stimulus group 1 and non-salient for stimulus group 2. Therefore, it is also possible that the non-salience of the correct stimulus during the acquisition stage or salience of the correct stimulus during the first reversal stage affected the ability of Fmr1 KOs in stimulus group 1 to reacquire the previously-learned response rule during the second reversal stage. Regardless of the underlying mechanism of the observed impairment in Fmr1 KOs, this impairment was dependent on stimulus salience because the group of Fmr1 KO mice tested under the opposite stimulus salience conditions was unimpaired on any reversal stage. Therefore, it is unlikely that Fmr1 KOs would have displayed an impairment if equally salient stimuli had been used.

4.2. Stimulus Salience

Relative salience of the two stimuli used in the experiment (see Figure 1A) was quantified by assessing stimulus preference on the first day of the acquisition stage. On the first day of a visual discrimination, mice are naive to the relationship between the visual stimuli and the reward. Therefore, biased responding to either of the stimuli on the first session of the acquisition stage of a visual discrimination indicates that the preferred stimulus is more salient than the non-preferred stimulus. In the present study, both Fmr1 KOs and wildtypes responded significantly more to the x stimulus than the + stimulus (see Figure 1B) on the first day of the acquisition stage, indicating that the x stimulus was the more salient of the two. Additionally, mice committed fewer errors to criterion on stages during which they were required to select the salient stimulus than on stages during which they were required to select the non-salient stimulus. With regard to this, Bussey et al. [51] have previously reported a stimulus bias in rats when discriminating between some, although not all stimulus pairs. Additionally, we have observed the same phenomenon in mice discriminating between simple stimuli (i.e., bias to a specific stimulus) and compound stimuli (i.e., bias to a dimension) in a touchscreen attentional set-shifting task (unpublished data).

In the present study, it may seem surprising that the two stimuli were not found to be equally salient considering that they were identical in overall size and brightness because the x stimulus was created by rotating the + stimulus 45 degrees. One possible explanation for this phenomenon is that mice were not attending to each stimulus as a whole, but instead were selectively attending more to certain regions of each stimulus. Regarding this, it seems reasonable that mice may have attended more to the lower region of each stimulus because the position of the stimuli on the screen was such that, when not rearing, the mouse’s line of sight was directed at the bottom of each stimulus rather than the top. In this scenario, it seems plausible that the x stimulus would be the more salient of the two stimuli because the bottom portion of the x stimulus is larger and brighter (i.e., two diagonal lines) than the bottom portion of the + stimulus (a single vertical line). Thus, the increased size and brightness of the lower region of the x stimulus may have increased the probability of approach to that stimulus or the ability to learn the relationship between that stimulus and the reward. However, it remains possible that factors other than stimulus salience underlie the observed stimulus bias.

4.3. Reward Collection Errors

During the acquisition stage, the percentage of trials during which Fmr1 KOs committed a reward collection error was similar to that of wildtypes (see Figure 3A). On the first reversal stage, the percentage of trials during which a reward collection error was committed decreased significantly in both Fmr1 KOs and wildtypes. However, this performance improvement was significantly greater in wildtype mice. Consequently, on the first reversal stage, the percentage of trials during which Fmr1 KOs committed a reward collection error was significantly greater than that of wildtypes. Between the acquisition stage and fourth reversal stage, the percentage of trials during which a reward collection error was committed declined to the same degree in Fmr1 KOs and wildtypes. However, although this performance improvement occurred rapidly in wildtypes, it was significantly more protracted in Fmr1 KOs (see Figure 3B).

There are several possible explanations for the increased number of reward collection errors committed by Fmr1 KOs during the first reversal stage and the protracted decline in reward collection errors committed by Fmr1 KOs across reversal stages. First, it is possible that the impaired performance of Fmr1 KOs represents a form of perseveration. Specifically, during the acquisition stage, mice learned to nosepoke the correct stimulus and then move to the rear of the chamber in order to collect a reward. When mice selected the incorrect stimulus, the house light was turned off and the dipper arm was not activated, signaling that no reward was available. Mice learned to recognize these stimuli and respond accordingly as evidenced by significantly fewer reward collection errors (following an incorrect stimulus choice) relative to reward collections (following a correct stimulus choice) during the acquisition stage and all four serial reversal stages (see Figure 3A).

That the number of reward collection errors and reward collections committed by Fmr1 KOs during the acquisition stage did not differ from the number committed by wildtypes suggests that Fmr1 KOs were unimpaired in their ability to learn (1) to collect a reward following a correct stimulus nosepoke and (2) that a reward was not available following an incorrect stimulus nosepoke. However, the significantly greater number of reward collection errors committed by Fmr1 KOs relative to wildtypes during the first reversal stage suggests that, following a reversal of the response rule learned during the acquisition stage, Fmr1 KOs perseverated in their attempts to collect a reward following a nosepoke to the previously-correct visual stimulus.

A second possibility is that this behavior represents the protracted extinction of a learned motor response. Specifically, during the two training stages immediately preceding the acquisition stage, a nosepoke to any stimulus on the touchscreen was rewarded. Thus, before entering the acquisition stage, mice had learned to move to the rear of the chamber to collect a reward following every nosepoke to a stimulus on the touchscreen. However, once mice entered the acquisition stage, they were only rewarded for making a nosepoke to the correct stimulus. Thus, the increased percentage of trials during which Fmr1 KOs made a head entry into the food receptacle during a timeout could represent the protracted extinction of the behavior learned during the training stages which preceded the acquisition stage (i.e., move to the rear of the chamber to collect a reward following every stimulus nosepoke). However, this possibility seems less likely because the difference between Fmr1 KOs and wildtypes did not appear until the first reversal stage. If the behavior observed in Fmr1 KOs reflects protracted extinction, differential performance of Fmr1 KOs and wildtypes would appear and be greatest during the acquisition stage.

It is also possible that the increased number of reward collection errors during the reversal stages was due to a general increase in activity in Fmr1 KOs, leading to an increased overall number of head entries. However, this possibility seems less likely because the differential performance of Fmr1 KOs and wildtypes did not occur during the acquisition stage but instead first appeared and was maximized during the first reversal stage. If the increased number of reward collection errors were simply due to a generalized increase in activity, differential performance of Fmr1 KOs and wildtypes would have been observed during the acquisition stage. Additionally, Fmr1 KOs and wildtypes did not differ at any stage on (1) latency to collect or attempt to collect a reward or (2) the percentage of reward collections during the 10 second reward period following a correct stimulus selection. A difference on these variables also would be expected if there were a general increase in activity in Fmr1 KOs.

Finally, it remains possible that Fmr1 KOs, following an incorrect stimulus choice, were prematurely attempting to initiate the next trial as opposed to attempting to collect a reward. However, if this behavior occurred, it did so exclusively following an incorrect stimulus choice because the percentage of trials on which mice made a food receptacle head entry following a correct choice did not differ between Fmr1 KOs and wildtypes at any stage.

4.4. Potential Mechanisms

In the present study, Fmr1 KOs displayed two behavioral deficits relative to wildtypes. First, Fmr1 KOs committed significantly more stimulus choice errors during the second reversal stage, but only when the reversal was difficult (i.e., non-salient correct stimulus, salient incorrect stimulus). A separate analysis of perseverative and learning errors revealed that increased errors did not occur during sessions immediately following the rule reversal when mice were still responding according to the previously-correct response rule, but instead occurred during later reversal sessions once mice had stopped responding using the old rule and had begun to respond using the new rule. These errors are similar to those committed by rats [43, 44] and mice [42] with experimentally induced excitotoxic lesions in the medial PFC (mPFC). In these studies, increased learning-type errors following mPFC lesion were interpreted as indicating a deficit in selective attention. Specifically, mPFC lesioned animals committed more errors at the learning stage of a reversal following a visual discrimination because they were less able than controls to attend to the relevant attributes of the visual stimuli. With regard to the potential role of the PFC in reversal learning performance, it should be noted that a double dissociation has been observed between (1) the late phase reversal learning impairment which results from mPFC lesion and (2) the early phase reversal learning impairment which results from orbitofrontal cortex lesion [43]. Thus, with regard to the underlying mechanism of the reversal learning impairment observed in the present study, one possibility is that the greater number of learning errors committed by Fmr1 KO mice was caused by the PFC neuropathology which has been observed in these mice [56]. For example, relative to wildtypes, Krueger et al. [56] reported decreased levels of multiple proteins in the PFC that are involved in synaptic function (e.g., NMDA receptor subunits NR1, NR2A, and NR2B) as well as reduced expression of c-Fos in Fmr1 KO mice.

In addition to committing significantly more stimulus choice errors during the late phase of the second reversal stage, Fmr1 KOs committed significantly more reward collection errors during the first reversal stage. One possibility is that these errors were due to greater behavioral inflexibility in Fmr1 KOs relative to wildtypes. Specifically, Fmr1 KOs may have persisted in checking for a reward following an incorrect stimulus choice despite visual and auditory stimuli (i.e., house light off and no sound from the dipper motor) indicating that the choice was incorrect and that no reward was available. A second possibility is that these errors reflect greater impulsivity in Fmr1 KOs relative to wildtypes. Specifically, Fmr1 KOs may have been prematurely attempting to initiate the next trial as opposed to attempting to collect a reward. Both behavioral inflexibility and impulse control have been linked to regions of the PFC [57] suggesting that one possible cause of the increased reward collection errors observed in the present study is the PFC neuropathology which has been observed in Fmr1 KO mice [43, 56].

It has recently been recognized that in addition to the PFC, the cerebellum is also involved in executive control [58], likely via it’s reciprocal connections to the PFC [59]. These findings are directly relevant to the executive function deficits observed in FXS because cerebellar pathology is present in both Fmr1 mice and humans with FXS [60]. With regard to Fmr1 KO mice, deficits include volume reductions of the nucleus interpositus and the fastigial nucleus in the deep cerebellar nuclei [61] and elongated spines in cerebellar Purkinje cell fibers as well as enhanced LTD induction at the parallel fiber synapses that innervate these spines [62]. Thus, it is possible that the deficits observed in the present study were the result of cerebellar pathology in Fmr1 KO mice. This hypothesis is consistent with recent findings indicating that stimulation of the dentate nucleus in the cerebellum evokes dopamine release in the mPFC and that this release is attenuated in Lurcher mutant mice exhibiting a total loss of cerebellar Purkinje cells [63]. More recent findings indicate that efferent pathways from the cerebellum modulate mPFC dopamine release via the thalamus and the ventral tegmental area [64] and that attenuation in the strength of cerebellar modulation of mPFC dopamine in Fmr1 KO and Lurcher mutant mice is accompanied by a functional reorganization of these cerebellar-PFC pathway [65]. Finally, it is important to note that in addition to the PFC and cerebellum, the striatum and thalamus have been implicated in executive dysfunction [21, 22]. Thus, it is possible that pathology in these brain regions may have contributed to the behavioral deficits observed in the present study.

5. Conclusion

In the present study, we assessed visual discrimination and serial reversal learning in Fmr1 KO mice and their wildtype littermates using a touchscreen operant conditioning task. Serial reversal learning performance was assessed under both difficult conditions (i.e., non-salient correct stimulus, salient incorrect stimulus) and easy conditions (i.e., salient correct stimulus, non-salient incorrect stimulus) to test the hypothesis that executive dysfunction in Fmr1 KO mice is only evident under conditions of high cognitive load. We observed two deficits in Fmr1 KOs relative to wildtypes. First, Fmr1 KOs committed significantly more errors during the second reversal stage, but only on the difficult reversal condition. Differential performance between Fmr1 KOs and wildtypes occurred late in the reversal stage once mice had begun responding using the new rule. This performance deficit was likely caused by an impairment in the ability to reacquire a previously-learned response rule which occurred only under conditions of high cognitive load. Second, during the first reversal stage, Fmr1 KOs committed significantly more attempts to collect a reward during the timeout following an incorrect response. This performance deficit may have been due to an impaired ability to respond flexibly following rule reversal or impaired impulse control. Previously reported findings of pathology in the PFC and cerebellum of Fmr1 KO mice may underlie the observed behavioral deficits.

Highlights.

• We assessed visual discrimination and serial reversal learning in Fmr1 KO mice.

• Cognitive load was manipulated by adjusting relative salience of the stimuli.

• Fmr1 KOs exhibited impaired selective attention under high cognitive load.

• Fmr1 KOs exhibited behavioral inflexibility which was unrelated to cognitive load.

• These deficits may be related to neuropathology in the cerebellum and PFC.