Individual differences reveal limited mixed-category effects during a visual working memory task

Abstract

Using stimuli from different categories may expand the capacity limits of working memory (WM) by spreading item representations across distinct neural populations. We explored this mixed-category benefit by correlating individuals’ behavioral performance with fMRI measures of category information during uniform- and mixed-category trials. Behaviorally, we found weak evidence for a mixed-category benefit at the group-level, although there was a high degree of individual variability. To test whether distinct neural patterns elicited superior performance in some individuals, we correlated a multivariate measure of neural category information with multiple behavioral metrics. This revealed a widespread positive relationship, intuitive for hit rate and working memory capacity, but counterintuitive for false alarm rate. Overall, these data suggest that mixed-category effects may support working memory performance, but unexpectedly, not all participants show this benefit. Only some people may be able to take advantage of representing mixed-category information in a differentiable way.

Visual working memory (WM) allows us to maintain a tightly limited amount of visual information over brief delays. These limitations functionally restrict WM capacity to ~4 items (Cowan, 2001). Given the importance of WM to daily life, understanding where capacity limitations arise is crucial (Engle, Tuholski, Laughlin, & Conway, 1999; Fallon, Zokaei, & Husain, 2016; Unsworth, Fukuda, Awh, & Vogel, 2014a, 2014b). A broad body of research explores factors contributing to capacity limitations at each stage of WM: encoding, maintenance, and retrieval (recent reviews of WM include D’Esposito & Postle, 2015; Eriksson, Vogel, Lansner, Bergstrom, & Nyberg, 2015; Lee & Baker, 2016). Several encoding-related factors are known to play a role, including attentional resources (Gazzaley & Nobre, 2012; Peterson et al., 2014), stimulus complexity (Alvarez & Cavanagh, 2004), stimulus similarity (Awh, Barton, & Vogel, 2007; Conrad & Hull, 1964), stimulus saliency (Melcher & Piazza, 2011), and set size (Bays, Catalao, & Husain, 2009; Fukuda, Awh, & Vogel, 2010; Gurariy, Killebrew, Berryhill, & Caplovitz, 2016).

At a more theoretical level, more than fifty years of cognitive psychology research have probed the architecture of cognition, including the interactions of perception, WM and episodic memory (e.g., Atkinson & Shiffrin, 1968). Key findings arise from dual-task experiments in which participants concurrently retain two sets of stimuli and make two responses. Measurements of inter-task interference provides insight regarding domain-specific (dependent on stimulus type) and domain-general (independent of stimulus type) operations. For example, greater verbal-verbal task interference compared to verbal-visual interference lead to the separate domain-specific visuospatial sketchpad and phonological loop and a domain-general central executive in the influential multicomponent model (e.g,. Baddeley, 1986; Baddeley, 2000; Baddeley & Hitch, 1974; Brooks, 1967; Cocchini, Logie, Della Sala, MacPherson, & Baddeley, 2002; Repovs & Baddeley, 2006; for a review of domain-specific and -general processing see Vandierendonck, 2016) (but see Morey, 2018). Another prominent view, the embedded processes model (Cowan, 1999), proposes some domain-specific WM storage within peripheral storage, and domain-general storage within the focus of attention (Cowan, Saults, & Blume, 2014; see also Larocque, Lewis-Peacock, & Postle, 2014). Traditional neuroimaging analyses have identified lateralized frontoparietal domain-specific WM storage, and bilateral frontoparietal domain-general activity showing heightened responses as set size increases (Chein & Fiez, 2010; Todd & Marois, 2004; Xu & Chun, 2006). Recent contributions from multivariate analyses have turned their focus toward the information available in ventral occipitotemporal regions after viewing visuospatial stimuli as an additional mechanism for domain-specific sensory storage (Ester, Serences, & Awh, 2009; Harrison & Tong, 2009; for a skeptical view of the sensory reactivation hypothesis see Xu, 2017).

Extending this work from the classic dorsal stream networks into category-selective ventral occipitotemporal regions confirms that above-chance classification follows imagining recently presented scenes (Johnson & Johnson, 2014), and preferential classification in face- and scene-selective regions (Han, Berg, Oh, Samaras, & Leung, 2013). In other words, there was significantly above chance classification in scene responsive areas when viewing or imagining a scene. It follows that WM trials distributing neural activity across different category-specific regions might support WM performance better than when the stimuli fall within the same category. Support for this view comes from a recent paper showing little interference in WM when the stimuli were visual shapes and visuospatial stimuli believed to have distinct neural representations (Sanada, Ikeda, & Hasegawa, 2015).

Recently, Cohen et al. (2014) reported superior WM performance after a clever stimulus manipulation in which they investigated a mixed-category effect using more distinct stimuli known to drive distinct neural populations (e.g., objects, scenes, bodies, faces). Performance was superior on mixed-category trials using stimuli drawn from multiple categories (e.g., bodies and faces) compared to uniform-category trials using stimuli drawn from a single category (e.g., bodies or faces). Furthermore, multivariate pattern classification revealed that greater neural separability between categories was associated with larger performance benefits for mixed-category trials. The authors proposed that this mixed-category benefit reflects the contribution of neural competition between simultaneously presented stimuli (e.g., Desimone & Duncan, 1995; Kastner & Ungerleider, 2000; Shapiro & Miller, 2011) to WM capacity limitations. They suggested that distributing neural activity across different category-selective regions facilitated WM, possibly by reducing stimulus interference.

Their observations raise questions regarding whether some proportion of variability in participants’ WM capacity could be attributed to inefficient neural separation. One atypical aspect of the Cohen et al. (2014) study was that it used a between-subjects design comparing behavioral performance in one set of participants (N = 60) with neural pattern separability in a second set of participants (N = 6). This precluded an analysis of how neural separability during different task phases (encoding, maintenance, retrieval) predicted behavioral performance. It also prevented probing the strength of the neural-behavioral correlations across participants. Participants with distinct category-related activation patterns would perform WM tasks more easily and exhibit a larger mixed-category benefit. In contrast, participants with more overlapping category-related activation patterns would perform worse and show a weaker mixed-category benefit. Furthermore, there is great interest in stabilizing or improving WM, and identifying the roots of individual differences can provide new targets for training paradigms.

This study investigated whether individual differences in WM are partially attributable to the ability to take advantage of different stimulus categories. We conducted an fMRI study in which participants performed a WM change detection task with memoranda from a single category (objects OR bodies) or from a mix of categories (objects AND bodies). We evaluated individual differences in behavioral performance and in the neural representations of object categories using a within-subjects analysis, separately for each phase of the WM task (encoding, maintenance, and retrieval). Our results revealed only modest support for the mixed-category benefit hypothesis. We confirmed that in some participants mixing categories improves WM performance and this was associated with greater neural category separability across WM task phases. However, our within-subjects analyses revealed a high degree of individual variability, with some individuals showing an impairment on mixed-category trials. While these behavioral differences were related to individual differences in neural category separability, our results suggest greater heterogeneity of mixed-category effects across participants than previously suggested.

Materials and Methods

Participants

Twenty-four participants (11 females, 20–38 years old) gave written informed consent prior to participating and received $50/hour. All observers were neurotypical, right-handed, reported normal or corrected-to-normal vision. All protocols were approved by the University of Nevada Internal Review Board. Participants completed two scanning sessions, one to collect high-resolution anatomical images for cortical reconstruction and one for the main experimental task.

Visual Display

The stimulus computer was a 2.53 GHz MacBook Pro with an NVIDIA GeForce 330M graphics processor (512MB of DDR3 VRAM). Stimuli were generated and presented using the Psychophysics Toolbox (Brainard, 1997; Pelli, 1997) for MATLAB (Mathworks Inc., Natick, MA) on a 32’ SensaVue (85 Hz refresh rate) visual display system (Invivo, Inc., Gainesville, FL) outside of the scanner bore and viewed with a tangent mirror attached to the head coil, permitting a viewable region of 31.5° × 18.9°. Stimulus presentation was time-locked to functional MRI (fMRI) acquisition via a trigger from the scanner at the start of image acquisition.

MRI Scanning Parameters

For all 24 participants, functional scans during the visual WM task were acquired at the Neuroimaging Facility of Renown Regional Medical Center in Reno, NV on a 3T Philips Ingenia scanner (software version 4.1.1) using a 32-channel digital SENSE head coil (Philips Medical Systems, Best, Netherlands). Functional images were obtained using T2* fast field echo, echo planar functional images (EPIs) sensitive to BOLD contrast (32 axial slices, 3.0 mm² in-plane voxel resolution, matrix = 80 × 80, slice-thickness = 2.5 mm, 1 mm gap, interleaved slice acquisition, FOV = 240 × 240, TE = 40 ms, TR = 2 s, flip angle = 71°).

Anatomical data were acquired at two imaging sites. For 17 participants, high-resolution anatomical images (MPRAGE: 208 sagittal slices, 0.9 mm² in-plane voxel resolution, matrix = 256 × 256, slice-thickness = 0.95 mm, FOV = 243 × 243 × 187 mm, TE = 4.33 ms, TR = 10 ms, flip angle = 7°) were acquired at the Renown Neuroimaging Facility. For the remaining 7 participants, high-resolution anatomical images (MPRAGE: 208 sagittal slices, 0.9 mm² in-plane voxel resolution, matrix = 256 × 256, slice-thickness = 0.95 mm, FOV = 243 × 243 × 187 mm, TE = 4.33 ms, TR = 10 ms, flip angle = 7°) were acquired at the University of California, Davis Imaging Research Center on a 3T Skyra MRI System (Siemens Healthcare, Erlangen, Germany) using a 64-channel phased-array head coil.

Visual WM Task

During functional runs, participants performed a change detection WM task (Figure 1). Each trial consisted of three phases: encoding, maintenance, and retrieval. Catch trials included only the cue and encoding periods.

Figure 1:

Trial sequence for the visual WM task. Participants were instructed to maintain fixation on the central square. Participants’ task was to indicate with a button press if any of the objects presented during the retrieval phase differed from those presented during the encoding phase. The example trial shows a mixed-category trial in which an object change occurred. On uniform-category trials (not shown), all four objects in the encoding and retrieval periods were selected from the same object category. On catch trials (not shown), the trial ended after encoding, excluding maintenance and retrieval phases and the color of the inter-trial interval fixation spot was red. In the experiment, the display for each phase was the same size; here, the encoding and retrieval phases are enlarged for visibility.

Participants were instructed to maintain constant fixation on a central square (0.25°). Each trial began with a white fixation square cue (300 ms), followed by stimulus encoding (1 s). Four grayscale stimuli appeared with one in each quadrant (Figure 1). Each stimulus spanned ~5° of visual angle in its maximum dimension and was presented on a random noise background (5° square) composed of 144 grayscale pixels (0.42° square), each differing by a random level of ±50 from the level of the uniform gray (128) background. The noise background ensured that participants could not rely on any salient global outline features of the objects to perform the task.

After encoding, the stimuli and noise patterns disappeared followed by a variable-length maintenance period. The duration was chosen pseudo-randomly to last 1, 2, or 3 s, with an equal number of each delay per run. At retrieval a second stimulus array appeared (1 s). Retrieval was followed by a variable-length response period selected pseudo-randomly to last 3, 4, or 5 s, with an equal number of each duration per run. The cue for the next trial appeared immediately after the response period. Each experimental trial lasted between 6.3 and 10.3 s.

The participants’ task was to detect whether any of the stimuli changed from encoding to retrieval and make a button press with their left (‘no change’) or right (‘change’) index finger. The noise patterns always differed across the encoding and retrieval periods (Figure 1) so participants could not rely on them to complete the task. During the response period of experimental trials, the fixation spot changed from white to blue to indicate that participants should make a response. Responses were accepted from the onset of the retrieval array to the end of the response period. Importantly, participants were instructed to respond as accurately as possible, without emphasizing response speed. We therefore focused our behavioral analysis on accuracy metrics (hit rate, false alarm rate, and percent correct) rather than reaction time.

Catch trials were randomly interspersed with experimental trials to aid in the separation of the different task phases in the fMRI analysis (see below). On catch trials, the cue and encoding array were identical to experimental trials, but the maintenance delay and retrieval phases were excluded. Encoding was immediately followed by an inter-trial interval, during which the color of the fixation spot changed to red to indicate that participants should withhold a response. On catch trials, participants were not required to maintain stimuli in WM or to respond. Each catch trial lasted 4.3, 5.3, or 6.3 s (300 ms cue + 1 s sample + 3, 4, or 5 s inter-trial interval).

Stimuli were drawn from the categories of tools, nontool graspable objects, and bodies. There were two experimental trial types. In ‘uniform trials’, all stimuli were from one category. On ‘mixed trials’, two stimuli were selected from two categories (e.g., two tools, two bodies, as in Figure 1). These categories yielded large mixed-category performance benefits in Cohen et al. (2014; they used all pairwise combinations of objects, bodies, faces, and scenes). Stimuli were randomly positioned across the quadrants, with no consistent relationship between quadrant and category. On trials that contained a change between encoding and retrieval, the stimulus changed to another exemplar from the same category (e.g., a tool changed to a different tool; Figure 1). All stimuli were different objects or people, and thus target changes were not simply changes in viewpoint, parts, or low-level features. The full set of images used can be found in Supplemental Figure 1.

Each run contained 6 catch trials and 24 experimental trials. The experimental trials included 4 trials of each of the unique trial types described above (uniform tools, uniform nontools, uniform bodies, mixed tools-nontools, mixed tools-bodies, mixed nontools-bodies). Preliminary analysis of the behavioral data indicated that there were no behavioral differences between the tool and nontool object categories. Thus, we collapsed the tool and nontool categories and treated them as a single ‘object’ category to contrast with the body category. Therefore, our analysis considered two types of uniform trials (objects and bodies) and one type of mixed trials (objects-bodies). This also increased statistical power by including more experimental trials in each analysis, and matched our object categories with one of the comparisons reported by Cohen et al. (2014).

Collapsing the tool and nontool categories left 12 uniform objects, 4 uniform body, and 8 mixed objects-bodies trials per run (plus 6 catch trials). The 30 total trials in each run were presented in pseudorandom order. A 16-s period of fixation was included at the start and end of each run. Participants completed 8 (n = 1), 9 (n = 12), or 10 (n = 11) runs, each lasting 4 min and 32 s, totaling 192 to 240 experimental trials per participant. Although the total number of trials was an added source of inter-participant variability, we simply collected as much data as possible with each participant; the number of runs were not based on any performance metric analyzed at the time of data collection.

Behavioral Analysis

We first looked for behavioral differences in accuracy across uniform and mixed trials, following Cohen et al. (2014). For each participant, we compared performance on mixed-category and uniform-category trials using hit rate (HR), false alarm rate (FA), and working memory capacity K (Pashler, 1988; Rouder, Morey, Morey, & Cowan, 2011), $K=SS\times \frac{(HR-FA)}{(1-FA)}$, where SS represents set size (always 4 in thedefined as current study). For each metric, we quantified the mixed-category effect by subtracting performance on the uniform-category trials from performance on the mixed-category trials (HR_mix-unif, FA_mix-unif, and K_mix-unif; e.g., HR_mix-unif = HR_mixed – HR_uniform). These metrics quantified any potential behavioral benefit on the WM task for trials using objects from multiple categories, rather than all from the same category (Cohen et al., 2014). For hit rate and working memory capacity, this metric is positive when performance was better on mixed-category trials. For false alarm rate, this metric is negative when performance was better on mixed-category trials.

At the group level, we used one-sample t tests to look for mixed-category effects across participants. However, we recognized that the effects of category structure might not be consistent across participants. Indeed, although some participants showed better performance on mixed-category trials (i.e., a mixed-category benefit), others showed worse performance on mixed-category trials (i.e., a mixed-category impairment). Thus, we additionally compared individual differences in the behavioral metrics with neural measures of category separability (see below).

fMRI Preprocessing and Analysis

High-resolution anatomical images were used to generate participant-specific surface reconstructions for each cortical hemisphere using FreeSurfer (Dale, Fischl, & Sereno, 1999; Fischl, Sereno, & Dale, 1999). Functional fMRI data were analyzed using AFNI (http://afni.nimh.nih.gov/afni/; Cox, 1996), SUMA (http://afni.nimh.nih.gov/afni/suma; Saad, Reynolds, Cox, Argall, & Japee, 2004), and MATLAB.

Functional scans were slice-time corrected to the first slice of every volume, motion corrected within and between runs, smoothed with a 6-mm Gaussian kernel, and normalized to percent signal change by dividing the voxel-wise time series by its mean intensity in each run. The BOLD response evoked by different phases of the WM task (encoding, maintenance, and retrieval) for each of the experimental trial types (uniform objects, uniform bodies, and mixed objects-bodies) was quantified in the framework of the general linear model (GLM; Friston et al., 1995), accounting for temporal autocorrelations of the noise (AFNI’s 3dREMLfit). Our event-related design with jittered timing for the maintenance and response periods and the inclusion of encoding-only catch trials allowed us to separately model regressors for the different trial phases. Square-wave regressors for each trial phase, for each condition, were generated and convolved with a response model (BLOCK model in AFNI’s 3dDeconvolve function for the encoding and retrieval periods, dmBLOCK model for the variable-delay maintenance period) accounting for the shape and temporal delay of the hemodynamic response. Nuisance regressors were included to account for variance due to baseline drifts across runs, linear and quadratic drifts within each run, and the six-parameter rigid-body head motion estimates. The GLM was run separately for odd and even runs, generating two independent estimates per regressor. Data from all trials were included in the GLM analysis and the analyses presented below; data from individual trials were not excluded for any reason.

The anatomical volume used for surface reconstruction was aligned with the motion-corrected functional volumes and the resulting transformation matrix was used to define a cortical mask for each hemisphere in volume space. Within this cortical mask, per hemisphere and individual participant, we ran a searchlight analysis (Kriegeskorte, Goebel, & Bandettini, 2006). We defined a spherical region of interest (ROI) around each voxel within the cortical mask, excluding all voxels within the defined radius but outside of the cortical mask (max of 256 voxels and ~12.3 mm radius). For each searchlight ROI, we compared the pattern of activity evoked by uniform object and uniform body trials using a correlation analysis across odd and even runs (Haxby et al., 2001). Specifically, we calculated Pearson correlations for uniform objects trials across the odd and even runs (r_obj:obj) and for the uniform bodies trials across the odd and even runs (r_bod:bod). These constitute the ‘within’ correlations, as they are independent measures of the same stimulus categories across odd and even runs. We also calculated the Pearson correlation for the uniform objects trials in the odd runs and the uniform bodies trials in the even runs (r_obj:bod), and vice versa (r_bod:obj). These constitute the ‘between’ correlations, as they are measures of different stimulus categories across odd and even runs. We computed a category information index (Δz) by subtracting the average Fisher transformed (z = 0.5 * log_e((1+r)/(1−r)) between-correlations from the average Fisher transformed within-correlations:

$$\Delta z=\left(z_{\text{obj:obj}}+z_{\text{bod:bod}}\right)/2-\left(z_{\text{obj:bod}}+z_{\text{bod:obj}}\right)/2$$

A positive Δz indicates that the pattern of BOLD activity evoked by objects and bodies were distinct and consistent across the independent odd and even runs. In short, the local region encodes consistent information that distinguishes between the stimulus categories.

We ran a complementary analysis for each trial phase (encoding, maintenance, retrieval), separately for each cortical hemisphere. Category information indices (Δz) were assigned to the central voxel of each searchlight. To compare the results across participants (i.e., group-level whole-brain analysis), each individual’s reconstructed cortical surface was warped to a standard surface-based space (Fischl et al., 1999) and then resampled in SUMA using an icosahedral shape to generate a standard mesh with a constant number (198,812) of co-registered nodes (Argall, Saad, & Beauchamp, 2006). Category information indices (Δz) were projected from each individual’s voxel space to the standard cortical surface for group-level analysis and visualization.

First, we generated statistical maps to identify regions in which local BOLD patterns could reliably distinguish between objects and bodies. We ran a one-sample t test against zero on the Δz category information metric, separately for each trial phase (encoding, maintenance, retrieval).

Second, we generated statistical maps to identify regions in which differences in local BOLD patterns across categories were correlated with behavioral differences in accuracy across uniform and mixed trials. We computed the Spearman correlation (r_behav:neural) between category information (Δz) and the mixed-category effects (HR_mix-unif, FA_mix-unif, and K_mix-unif) across participants for each node in the standard cortical surfaces. We computed a separate correlation for each behavioral metric. For this analysis, we only considered nodes that showed significant category information, leading to a conjunction map of nodes that reached statistical significance (p < .05, uncorrected) for both the category information (Δz) and the neural-behavioral correlation (r_behav:neural) analyses (p < .0025, uncorrected). In other words, the regions identified as carrying significant category information served as a mask for our analysis of the correlation between the neural and behavioral metrics (see below). Finally, using Monte Carlo simulations to maintain a family-wise error rate of α = .05, we implemented a surface-based cluster correction (minimum cluster size of 112 mm² for left hemisphere, and 149 mm² for right hemisphere). The simulations used the AFNI scrips slow_surf_clustsim.py to randomly generate Gaussian noise in volume space, projected to a cortical surface mesh and smoothed to a level that exceeded (by ~2.78 times) the average estimated smoothness of the residuals from the GLM. These random data iterations were used to estimate the surface area cluster size associated with the desired family-wise error rate.

We note that we applied liberal thresholding in an attempt to replicate the previously reported between-subject association between neural category separability and the behavioral benefits of mixed-category trials on WM (Cohen et al., 2014) in our within-subject design.

Results

Group-Level Behavioral Performance

Participants performed a visual WM change detection task (Figure 1). Overall accuracy was 80.0% ± 6.2 (M ± SD), ranging between 68.8% and 92.7% across individuals. Mean reaction times were 1490 ms (SD = 340), ranging between 742 and 1962 ms across individuals. As accuracy was emphasized in the task instructions, we focus our analysis on three related metrics: hit rate (HR), false alarm rate (FA), and working memory capacity (K). These performance metrics showed good inter-participant reliability, as quantified by split-half (odd vs. even runs) correlations: hit rate (r(22) = .70, p < .001, R² = .49), false alarm rate (r(22) = .66, p < .001, R² = .44), and K (r(22) = .59, p = .003, R² = .35). In other words, participants tended to perform fairly consistently throughout the session.

False alarm rates were generally low (12% ± 9, range = 0.8 to 33%) and hit rates were generally high (72% ± 12, range = 48 to 92%). The mean WM capacity (K) was 2.7 (SD = 5.1), with a range of 1.7 to 3.6. Across participants, accuracy was highly correlated with hit rate (r(22) = .82, p << .001, R² = .67) and working memory capacity (r(22) = .88, p << .001, R² = .77). The results reported below for hit rate and working memory capacity were qualitatively similar to those for accuracy.

We first compared performance on mixed trials (including objects and bodies) and uniform trials (including objects or bodies) for the three behavioral metrics. Neither the K_mix-unif metric (M = 0.046, 95% CI [−0.097 0.189]), t(23) = 0.66, p = .51, nor the HR_mix-unif metric (M = −2.18, 95% CI [−5.95 1.59]), t(23) = −1.19, p = .25, were significantly different from zero. The FAmix-unif metric (M = −3.33, 95% CI [−5.51 −1.14]) was significantly less than zero, t(23) = −3.15, p = .005, ω² = 0.27, indicating that FA rates were lower for mixed trials compared to uniform trials across our participants. The false alarm results provide some evidence of a mixed-category benefit, supporting the view that perceptual encoding and/or WM maintenance and retrieval can more readily handle multiple stimuli from different object categories. However, the null results for hit rate and working memory capacity suggest a weak mixed-category benefit, on average.

Individuals who show a mixed-category benefit may have more distinct neural representations of objects and bodies than individuals without a category benefit or a mixed-category impairment. In short, a participant who isolates neural responses to a particular neural population should demonstrate superior WM performance because the WM representation should be more distinct and less prone to interference from other stimuli. Below, we explored individual differences in the relationship between behavioral performance (using all three metrics) and category separability across brain regions. First, it is worth noting the relationship between the three behavioral metrics. Across participants, HR_mix-unif and FA_mix-unif were modestly correlated (r(22) = .38, p = .07, R² = .14), consistent with a tradeoff between hits and false alarms due to different response criterion across participants. Although WM capacity was computed from hit rates and false alarm rates, K_mix-unif was highly correlated with HR_mix-unif (r(22) = .98, p << .001, R² = .96), but not FA_mix-unif (r(22) = .24, p = .24), indicating that our measure of WM capacity was driven more by hits than false alarms.

Category Information in Neural Activity Patterns

To quantify category separability at the neural level, a searchlight analysis compared local patterns of BOLD activity evoked by uniform-category trials across odd and even runs and for each stage of the task (Figure 3). We defined a category information index (Δz) as the difference of the within-category correlations and the between-category correlations. Positive values indicate that patterns of BOLD activity consistently distinguished between objects and bodies. This analysis served as a localizer; we restricted further analyses to those regions showing significant category separability. We show the results of this analysis using a liberal threshold as it is the same mask threshold used for the behavioral-neural correlation analysis, below.

Figure 3:

Multivariate analyses demonstrate robust category representations during the WM task. Group-level searchlight analysis of category information (Δz) across participants during encoding (A), maintenance (B), and retrieval (C). This analysis was restricted to uniform-category trials, in which stimuli from a single category (body or objects) was present. Hot colors represent regions in which category information was significantly greater than zero (p < .05, uncorrected), indicating that body and object stimuli evoked consistent patterns of BOLD activity in these regions. A selection of major sulci for these views are depicted in Figure 4.

Objects and bodies evoked distinct patterns of BOLD activity in a wide range of cortical regions of both hemispheres during all three phases of the WM task (Figure 3). As the statistical power of our analyses limits conclusions about individual brain regions, we focus our interpretation on the overall patterns of significance. As expected, during encoding and retrieval (Figure 3, top and bottom), when stimuli were visible, category information was prominent in occipital and ventral temporal regions, with pockets of significance in frontal regions. During maintenance (Figure 3, middle), significant category information was most prominent in the left inferior parietal lobe, with smaller areas of significance in ventral temporal and frontal cortex.

As Δz is a difference score, it is possible that the positive correlations shown in Figure 3 were derived from negative within- and between-category correlations. Although some negative between-category correlations are expected (they should be zero ± some error), negative within-category correlations would be nonsensical. We verified this was not the case. Very few nodes carrying significant category information (Δz > 0; Figure 3) yielded negative within-category correlations (between 0% and 0.42%; n = 0 to 147). In other words, the category information localizer shown in Figure 3 and used as a conjunction mask in the remaining analyses was almost exclusively composed of nodes with the expected positive within-category correlations. Removal of nodes with negative within-category correlations from the further analyses did not alter the results or the interpretation of the data.

Individual Differences: Correlation of Behavioral and Neural Mixed-Benefit Effects

Differential performance across mixed and uniform-category trials represents the mixed-category effect. The searchlight analysis quantified the degree of neural category separation for BOLD patterns across cortex. We next determined if the behavioral and neural results were correlated across participants, extending the analysis reported by Cohen et al., (2014). It directly tests their hypothesis that superior mixed-category performance stems from more distinct neural patterns.

For each node and participant and task phase, we correlated the category information index (Δz) with each behavioral metric (FA_mix-unif, HR_mix-unif, K_mix-unif). We limited consideration to nodes showing significant category information at the group level, yielding a conjunction (p < .0025, uncorrected) of the behavior-neural correlation (p < .05) and category information (p < .05) results (Figure 4, FA_mix-unif; Figure 5, HR_mix-unif; Figure 6, K_mix-unif). To reiterate, nodes had to show significant Δz (Figure 3) and be correlated across participants with the corresponding behavioral metric of the mixed-category effect.

Figure 4:

Behavioral-neural correlations reveal that greater separability in neural patterns for objects and bodies was generally positively correlated with individual differences in false alarm rates across mixed and uniform-category trials. This was unexpected, as it suggests that more distinct neural patterns across categories were associated with worse behavioral performance on mixed-category trials. Behavioral-neural correlations (r_behav:neural) were calculated between category information (Δz) and FA_mix-unif separately for encoding (A), maintenance (B), and retrieval (C). The clusters represent regions in which the conjunction of r_behav:neural (p < .05, uncorrected) and category information (Δz; p < .05, uncorrected; see Figure 3), were robust (p < .0025, surface-based cluster correction to family-wise error α = 0.05). Hot colors represent significant positive r_behav:neural values. Several major sulci are labeled for reference: calcarine sulcus (calc), cingulate sulcus (cing), collateral sulcus (cos), central sulcus (cs), intraparietal sulcus (ips), postcentral sulcus (pcs), parietooccipital sulcus (pos), superior temporal sulcus (sts).

Figure 5:

Behavioral-neural correlations reveal that greater separability in neural patterns for objects and bodies was generally positively correlated with individual differences in hit rates across mixed and uniform-category trials. This is consistent with the hypothesis that more distinct neural patterns across categories are associated with better behavioral performance on mixed-category trials. Behavioral-neural correlations (r_behav:neural) were calculated between category information (Δz) and HR_mix-unif, separately for each phase of the trial: encoding (A), maintenance (B), and retrieval (C). Conventions and abbreviations are the same as in Figure 4.

Figure 6:

Behavioral-neural correlations reveal that greater separability in neural patterns for objects and bodies was generally positively correlated with individual differences in working memory capacity across mixed and uniform-category trials. This is consistent with the hypothesis that more distinct neural patterns across categories are associated with better behavioral performance on mixed-category trials. Behavioral-neural correlations (r_behav:neural) were calculated between category information (Δz) and K_mix-unif, separately for each of the three phases of the visual WM task: encoding (A), maintenance (B), and retrieval (C). Conventions and abbreviations are the same as in Figure 4.

The correlation between false alarm rate (FA_mix-unif) and category information at the neural level (Δz) was largely positive (75% of clusters). Participants showing better neural discrimination between objects and bodies showed higher false alarm rates (i.e., worse performance) for mixed-category trials. Surprisingly, although the group behavioral effects on false alarm rates showed better performance on mixed-category trials (Figure 2), individual differences in false alarm rates were correlated with neural measures of category separability in the opposite direction.

Figure 2:

Behavioral data. Solid black circles represent the mean difference between mixed-category and uniform-category trials (mix-unif) for each behavioral metric (HR = hit rate; FA = false alarm rate; K = working memory capacity). Black lines represent the 95% CI. Light gray circles represent each of the 24 participants, with random horizontal jitter to facilitate visualization.

Positive correlations were also present between category information at the neural level (Δz) and hit rate (HR_mix-unif, 90% of clusters), and working memory capacity (K_mix-unif, 90% of clusters). Here, participants who had more separable neural representations of category performed better on mixed-category trials.

Discussion

One goal of the current work was to determine whether a mixed-category benefit could be leveraged to strategically enhance WM capacity. The logic is that stimuli from several discriminable categories associated with specialized neural populations in object-selective cortex elicit superior WM performance by distributing neural activity across multiple regions. In other words, mixed-category trials spread the WM load across a larger pool of neurons compared to uniform-category trials. Our inspiration was to follow up on work demonstrating a mixed-category benefit in a large group of behavioral participants and associating it with more distinct neural patterns in a separate, smaller group of fMRI participants (Cohen et al., 2014), and work revealing detectible classification differences in category-selective regions during WM tasks (Han et al., 2013; Johnson & Johnson, 2014; Sanada et al., 2015). What we observed was more complex and nuanced, as is often the case.

Behaviorally, we observed lower false alarm rates for the mixed-category condition. No mixed-category benefit emerged in hit rate or working memory capacity. This group-level analysis of our behavioral data suggests that the mixed-category effect was inconsistent across participants. One possibility is that mixed-category effects may contribute to working memory capacity, but in a very limited manner. Alternatively, some participants may benefit on mixed-category trials, whereas others do not, leading to high variability across participants. Indeed, Figure 2 shows that this was the case.

To better understand the mixed-category effects across participants we looked at the correlation between behavioral and neural effects related to stimulus condition. Pattern classification revealed regions generally positively correlated with all three behavioral measures (FA, HR, K), across all phases of WM, and over a broad range of occipital, ventral temporal, inferior parietal, and lateral frontal regions. This analysis confirmed the expected pattern of better performance with higher stimulus category separability for hit rate and working memory capacity. This observation extends Cohen et al. (2014) by showing that stimulus category is relevant across encoding, maintenance, and retrieval. Unexpectedly, there was a positive correlation with false alarm rate such that high false alarm rates were correlated with high category separation. One testable possibility is that false alarm rates reflect each participant’s response criterion. Indeed, we observed that participants with higher false alarm rates also had higher hit rates.

Overall, our results demonstrate that a mixed-category benefit is not universal. Only some participants garnered this strategic benefit. The majority showed no mixed-category improvement, and some showed a mixed-category impairment. Thus, our results provide some support for the hypothesis provided by Cohen et al. (2014) that greater differentiation in the neural representation of task stimuli can enhance visual WM performance. However, even though we applied liberal statistical thresholds this support was weak. Some results were inconsistent with the mixed-category hypothesis (e.g., positive correlations between false alarm rate and neural separability). If the mixed-category effect were robust, we would expect to see stronger effects across measures and participants. Thus, we conclude that the mixed-category effect, though logically attractive, appears to be more heterogeneous than anticipated.

In thinking about why the current data does not offer compelling support of the mixed-category effect it is worth noting methodological differences between the current study and that of Cohen et al. (2014). We used fewer stimulus categories and applied a within-subjects imaging design. Yet another important difference is that they conducted their study at an elite academic institution raising the possibility that their participants were more homogeneous and/or more strategic at leveraging the mixed-category condition. At a large public institution, the student population is heterogeneous and is likely more variable in their use of advantageous WM strategies. Importantly, because strategy training can be effective (Mowszowski, Lampit, Walton, & Naismith, 2016), new protocols that encourage categorization distinction might serve to better distribute representations across different category-specific neural populations and improve WM performance. Alternatively, if a stimulus-based strategy emerged, it might be artefactual to the task design. Our participants completed all trials in the scanner, which may have impaired our participants’ performance because it is less comfortable and less arousing than the lab. At the same time, if mixed-category effects were robust, we would have expected them to hold up under these modified procedures. As stated above, this may indicate a more nuanced and heterogeneous mixed-category effect.

What explains the heterogeneity of responses to the mixed-category condition? We return to our speculation that WM strategy is importantly modulating how participants approach the task. It seems plausible, and testable, that the more attention paid to stimulus category, the more separable neural representations might be, thereby improving WM performance. An example of ways the task differentially engages cortical networks comes from our previous finding that WM tasks probed by recall or recognition correlate with different frontoparietal engagement. For patients with parietal lobe damage, WM performance was impaired on recognition tasks, but not those tested with recall (Berryhill, Chein, & Olson, 2011). An intriguing application of these data is that individual differences may be targets for future WM training protocols.

A final question is to ask whether these data contribute to our understanding of the cognitive architecture of WM as understood in theoretical models. As mentioned in the introduction, the tentative linkages between existing cognitive models (e.g., the multimodal model, embedded processes, etc.) and cognitive neuroscience are largely in frontoparietal networks. Emerging research over the last decade highlights the potential contributions from the sensory regions contribute to cognition in some way we are just beginning to understand (Roelfsema & de Lange, 2016; but see Xu, 2017). Many data points are missing as we begin to understand what role silent activity (Rose et al., 2016) and oscillatory mechanisms (Roux & Uhlhaas, 2014) play in WM. It is with some optimism that we suggest that answering these questions will advance our theoretical footing and supply new targets for stabilization and maintenance of WM. To address remaining questions regarding WM capacity limitations and the nature of the mixed-category effect, it will be important to design WM studies that permit and facilitate analyses across stimulus category and task phase, using sensitive behavioral measurements that include criterion.

Conclusions

Because WM is capacity limited, any manipulation that demonstrably expands WM capacity is of interest. Some participants may improve WM performance when holding on to items from different stimulus categories (e.g., bodies vs. objects). These participants have more distinct neural representations of stimulus category during all stages of WM and over a large expanse of cortex. However, our analysis revealed a relatively weak mixed-category benefit with a high degree of variability across individuals, only some of whom take advantage of a potential benefit on mixed-category trials. Thus, future work will be important to identify how generalizable this information is and whether this basic principle can be explicitly refined to better support WM.

Highlights:

• Mixed-category stimuli may be a useful strategy to improve WM performance

• High degree of individual variability in mixed-category benefit

• Behavior positively correlated with greater neural category separability

• Training to use category information in WM may be beneficial