Noradrenaline tracks emotional modulation of attention in human amygdala

Summary

The noradrenaline (NA) system is one of the brain’s major neuromodulatory systems; it originates in a small midbrain nucleus, the locus coeruleus (LC), and projects widely throughout the brain.^1,2 The LC-NA system is believed to regulate arousal and attention^3,4 and is a pharmacological target in multiple clinical conditions.^5,6,7 Yet our understanding of its role in health and disease has been impeded by a lack of direct recordings in humans. Here, we address this problem by showing that electrochemical estimates of sub-second NA dynamics can be obtained using clinical depth electrodes implanted for epilepsy monitoring. We made these recordings in the amygdala, an evolutionarily ancient structure that supports emotional processing^8,9 and receives dense LC-NA projections,¹⁰ while patients (n = 3) performed a visual affective oddball task. The task was designed to induce different cognitive states, with the oddball stimuli involving emotionally evocative images,¹¹ which varied in terms of arousal (low versus high) and valence (negative versus positive). Consistent with theory, the NA estimates tracked the emotional modulation of attention, with a stronger oddball response in a high-arousal state. Parallel estimates of pupil dilation, a common behavioral proxy for LC-NA activity,¹² supported a hypothesis that pupil-NA coupling changes with cognitive state,^13,14 with the pupil and NA estimates being positively correlated for oddball stimuli in a high-arousal but not a low-arousal state. Our study provides proof of concept that neuromodulator monitoring is now possible using depth electrodes in standard clinical use.

Highlights

• •Sub-second neuromodulator dynamics can be measured using clinical depth electrodes
• •Noradrenaline dynamics in human amygdala reflect attention and arousal
• •Coupling between noradrenaline and pupil dilation depends on cognitive state

Bang et al. demonstrate that electrochemical estimates of sub-second neuromodulator dynamics can be obtained in humans using clinical depth electrodes. They show that noradrenaline tracks the emotional modulation of attention and that the coupling between noradrenaline and pupil dilation—a common behavioral proxy for noradrenaline—depends on cognitive state.

Results

Electrochemistry on clinical depth electrodes

Recent years have seen the novel application of electrochemistry during awake deep brain stimulation (DBS) surgery, in which electrodes are implanted for the management of movement disorder symptoms (e.g., Parkinson’s disease).^{15,16,17,18,19} While providing the first estimates of fast neuromodulator release from the human brain, these recordings require custom-made carbon-fiber electrodes and have naturally been constrained to the basal ganglia, mainly dorsal striatum, by the surgical procedure. However, an accurate understanding of neuromodulatory systems in human health and disease requires an ability to study their function across the brain,^20,21 especially in the case of the locus coeruleus (LC)-noradrenaline (NA) system, which plays a minimal role in the regulation of striatal activity.²

Here, we present an approach for obtaining sub-second neuromodulator estimates from clinical depth electrodes that are implanted throughout the brain for phase II epilepsy monitoring.²² Patients with medication-resistant epilepsy can become eligible for ablation of epileptic foci. To help localize such foci, depth electrodes are implanted in brain regions relevant to each patient’s condition and neuronal activity is tracked over multiple days in an epilepsy monitoring unit (EMU). For years neuroscientists have utilized these electrodes for intracranial electrophysiology²³—we show that they can be used for intracranial electrochemistry, too.

The workflow is summarized in Figure 1A. In brief, the neurosurgeon implants depth electrodes (here, Ad-Tech macro-micro) into potential epileptic loci. During their stay in the EMU, the patient performs a research task while an electrode in a region of interest is used for voltammetric recordings. Once epilepsy monitoring is complete, the neurosurgeon explants the electrodes. A signal prediction model is then created by exposing the relevant electrode to labeled concentrations of NA, dopamine (DA), serotonin (5-HT), and pH in a controlled in vitro setting. Finally, to generate in vivo neuromodulator estimates, the signal prediction model is applied to the voltammetric recordings from the patient brain.

Figure 1.

Electrochemistry on clinical depth electrodes

(A) Sketch of electrochemical approach.

(B) We made electrochemical recordings in the amygdala of three patients while they performed the task shown in Figure 2A.

(C) In vitro evaluation of electrochemical approach. The macro-micro electrodes were explanted from the three patients who performed the task in Figure 2A. The Behnke-Fried electrodes were explanted from the amygdala of three patients at another hospital and were included to demonstrate generalizability. Dots indicate the average predicted NA concentration (nanomoles, nM) for single-analyte solutions that only contained NA (green), DA (blue), or 5-HT (red). Green diamonds indicate mixture solutions that contained DA and/or 5-HT in addition to NA. Predictions were from a 10-fold cross-validation and pooled across patients. R² values were obtained by regressing the predicted NA concentration against the true NA, DA, or 5-HT concentration. Error bars represent 95% confidence intervals but are not visible at this scale. See Figure S1 for in vitro evaluations for DA and 5-HT.

We applied this approach in three patients using electrodes implanted in the amygdala (Figure 1B), an evolutionarily conserved structure that supports emotional processing^8,9 and receives dense LC-NA projections.¹⁰ In vitro evaluation using out-of-training data showed that our approach can accurately detect NA and does not confuse NA with DA or 5-HT (Figure 1C, left). Our approach also generalized to another type of depth electrode explanted from the amygdala of three patients at another hospital (Ad-Tech Behnke-Fried; Figure 1C, right).

Visual affective oddball task

The three patients performed a visual affective oddball task (Figure 2A). The task involved the rapid presentation of unique images from the International Affective Picture System (IAPS)¹¹ as surprising (oddball) stimuli and a checkerboard image as the standard stimulus. The IAPS images were selected based on the content ratings provided by the IAPS and grouped within a block design such that every block contained images of the same emotional category. There were four emotionally evocative blocks, which varied in terms of valence (negative versus positive) and arousal (low versus high), and two emotionally neutral blocks. To keep patients engaged, they were told to press a button whenever an oddball stimulus was shown.

Figure 2.

Experimental framework and behavioral results

(A) Patients performed a visual affective oddball task while we simultaneously measured NA in the amygdala and pupil dilation with sub-second temporal resolution. The task was divided into six blocks of 100 images; the images were shown for 1 s and separated by 1 s blank intervals. Within each block, 80% of the images were a checkerboard image (standard) and the remaining 20% were unique IAPS images (oddball). See Figure S2A for IAPS content ratings and measured illuminance for the different image sets.

(B) Oddball PDR for (top) patients and (bottom) controls across all trials.

(C) Oddball PDR for (top) patients and (bottom) controls within each condition.

(B and C) For each trial, we smoothed the time series using a 0.5 s causal filter, Z scored the smoothed data and removed any linear drift. Lines indicate the difference between the average time series for oddball and standard trials. Triangles indicate the significance (p < 0.050) of a predictor and their direction indicates the sign of the effect (down, negative; up, positive). Only significant predictors are shown. See Figure S2B for an analysis in raw units, which also considers image illuminance and trial history.

In parallel to the electrochemical recordings, we monitored pupil dilation, a widely used behavioral proxy for activity in the LC-NA system.¹² The EMU setting meant that we did not have full control over extraneous variables, such as head position, eye movement, and room lighting, which can affect pupil dilation. We therefore also ran the task in a group of control subjects (n = 17) in a standard lab setting and used correspondence between the patient and the control data as an indicator of data quality.

Behavioral validation of experimental framework

To validate the pupil data from the EMU and the task, we first tested for an oddball pupil dilation response (PDR), by applying a linear mixed-effects regression in which we predicted the pupil estimate at each time point using stimulus type (standard = −1; oddball = 1). This analysis returned an oddball PDR: pupil dilation was larger for oddball than standard stimuli, and this difference was largest around 1.2 s after stimulus onset (Figure 2B, top). We next analyzed oddball trials only, now predicting the pupil estimates using emotional valence (negative = −1; neutral = 0; positive = 1), emotional arousal (low = −1; neutral = 0; high = 1), emotionally evocative (neutral = −1; evocative = 1), and an interaction between valence and arousal. The valence, arousal, and evocative terms were specified at the block level as per our task design (Figure 2A); valence and arousal were considered features of blocks involving emotionally evocative IAPS images only (Figure S2A), with the evocative term capturing any difference between these blocks and emotionally neutral blocks. This analysis identified a main effect of valence and an interaction between valence and arousal, with pupil dilation strongest for IAPS images characterized by negative valence and high arousal (Figure 2C, top). In further support of the quality of the pupil data collected in the EMU, these effects closely resembled those observed in the control group (Figures 2B and 2C, bottom). Finally, we examined button presses. Indicative of a high level of task engagement, patients detected nearly all oddball stimuli (patients, 94%–100%; controls, 96%–100%). An analysis of the impact of the task factors on reaction times did not return any effects in the patient group, but it identified an effect of arousal in the larger control group, with responses being slower in high-arousal blocks (Figure S2C).

NA estimates track the emotional modulation of attention

Our task was designed to probe attention-related processes, by randomly interspersing the repeated presentation of a checkerboard image with IAPS images, and to induce different cognitive states, by grouping the emotional content of the IAPS images within a block design (Figure 2A). To quantify the impact of these factors on NA dynamics, we employed a linear mixed-effects regression in which we predicted single-trial estimates of the NA response around stimulus presentation using stimulus type, emotional valence, emotional arousal, emotionally evocative, and interactions between these terms.

This analysis identified a positive main effect of arousal and a positive interaction between stimulus type and arousal (Figure 3A; arousal, t(1,790) = 3.34, p < 0.001; type × arousal, t(1,790) = 4.35, p < 0.001). To unpack these effects, we adopted a simple-effects approach, quantifying the impact of arousal for each stimulus type and the impact of stimulus type for each level of arousal. Indicating that NA is modulated by emotional content, we found that the estimated NA response for oddball stimuli was higher in high-arousal than low-arousal blocks (Figure 3B; effect of arousal for oddball stimuli, t(1,790) = 4.30, p < 0.001), whereas the estimated NA response for standard stimuli did not differ between these blocks (Figure 3B; effect of arousal for standard stimuli, t(1,790) = −1.13, p = 0.259). We note that the positive effect of arousal for oddball stimuli remained after controlling for reaction times in an analysis of oddball trials only (arousal, t(346) = 4.00, p < 0.001). Indicating that these effects reflect contextual modulation of the attentional salience of surprising stimuli, we found that the estimated NA response was higher for oddball than standard stimuli in high-arousal blocks but lower in low-arousal blocks (Figure 3B; effect of type for high arousal, t(1,790) = 2.26, p = 0.024; effect of type for low arousal, t(1,790) = −2.09, p = 0.037). While a stronger response for oddball than standard stimuli is in line with theories of LC-NA function, there is, to our knowledge, no account that predicts the opposite relationship. In addition to these arousal-related effects, the analysis identified a positive main effect of emotionally evocative (Figure 3A; evocative, t(1,790) = 1.98, p = 0.048), indicating that the estimated NA response was overall higher in emotionally evocative than emotionally neutral blocks.

Figure 3.

NA estimates track the emotional modulation of attention

(A) Regression coefficients ± SE from linear mixed-effects regression described in main text. ^∗p < 0.050; ^∗∗p < 0.010; ^∗∗∗p < 0.001.

(B) Single-trial NA estimates (mean ± SE) separated by emotional arousal and stimulus type.

(A and B) Single-trial NA estimates were calculated as the mean NA estimate over a 1 s window centered on stimulus onset minus the mean NA estimate over the preceding 0.5 s and Z scored across trials for each patient. See Figure S3 for analysis of DA and 5-HT estimates.

While not the focus of our study, our electrochemical approach also returned estimates of DA and 5-HT. Applying the same analysis as for NA, we found that the estimated DA response for oddball stimuli was higher in emotionally evocative than emotionally neutral blocks, whereas the estimated 5-HT response was overall lower in emotionally evocative than emotionally neutral blocks (Figure S3). The DA and 5-HT results help clarify the specificity of the NA results, with the estimated NA response reflecting contextual modulation of attention in the most emotion-specific manner.

Coupling between pupil and NA estimates varies with emotional arousal

We next turned to the relationship between the pupil and NA estimates. The presence of pupil-NA coupling would both support the use of pupillometry as an indirect measure of LC-NA activity¹² and corroborate our electrochemical approach. In addition to providing the first-ever characterization of the link between pupil dilation and NA in a key LC-NA projection target, these data may also help us understand how pupil-NA coupling changes with cognitive state.^13,14 We used the estimated pupil and NA time series across all trials to test for a general presence of pupil-NA coupling and—guided by the NA results—the time series for oddball trials in low- and high-arousal blocks to test for a dependence of pupil-NA coupling on cognitive state. For each condition of interest, we quantified the relationship between the estimated pupil and NA time series averaged across patients in two ways. First, we conducted simple (Pearson) correlations. Second, as simple correlations cannot capture time-varying relationships, we also fitted a hidden Markov model (HMM), a popular approach for unsupervised statistical learning of multivariate time series data that can identify changes in correlation without making a priori assumptions about when these changes might occur.^24,25

In support of pupillometry as an indirect measure of LC-NA activity and our electrochemical approach, we found a positive correlation between the estimated pupil and NA time series averaged across all trials (Figure 4, top left; r = 0.37, p = 0.004). In addition, in line with the hypothesis that pupil-NA coupling depends on cognitive state, we found a negative correlation between the average estimated pupil and NA time series for low-arousal oddball trials (Figure 4, top middle; r = −0.59, p < 0.001) but a positive correlation for high-arousal oddball trials (Figure 4, top right; r = 0.74, p < 0.001). Importantly, the HMM approach showed that the positive correlation across all trials was strongest (Figure 4, bottom left), and that the difference in the sign of the correlation between low-arousal and high-arousal oddball trials was largest (Figure 4, bottom middle versus bottom right), around stimulus presentation.

Figure 4.

Coupling between pupil and NA estimates varies with emotional arousal

The panels show (top) the average estimated pupil and NA time series with simple correlation statistics reported and (bottom) the HMM estimate of pupil-NA coupling for (left) all trials irrespective of stimulus type, (middle) oddball stimuli in low-arousal blocks, and (right) oddball stimuli in high-arousal blocks. We first smoothed the time series using a 0.5 s causal filter and Z scored the smoothed data for each trial. We then averaged the time series for each condition, removed any linear drift, and scaled the average time series to have unit variance, by dividing each time point by the average of the condition-specific standard deviations over the average time series. The last step helps ensure that estimated correlations, which are sensitive to the variance of the data, are comparable across conditions. Shaded area for the HMM estimate of pupil-NA coupling indicates 95% credible interval. See Figure S4 for simple correlations for oddball stimuli in each block.

Discussion

The LC-NA system affects neural activity across the brain.^1,2 This influence, often summarized as an impact on global brain states,^26,27 has been the target of theoretical models cast at different levels of description, including neuronal processes such as gain control and cognitive processes such as arousal and attention.^3,4,28,29 Clinically, the LC-NA system is implicated in multiple conditions, perhaps most prominently attention-deficit/hyperactivity disorder.^7,30,31,32 Yet there has not been a way to directly measure NA in humans. Here, we provide proof of concept that electrochemical estimates of sub-second NA dynamics can be obtained using clinical depth electrodes implanted for epilepsy monitoring.

By applying our electrochemical approach in human amygdala together with a visual affective oddball task and pupillometry, we tested theoretical predictions about the role of NA in arousal and attention and examined the relationship between pupil dilation and NA. In our task, IAPS images of a particular emotional category were shown on 20% of trials (oddball) and a checkboard image on the remaining 80% of trials (standard). In line with a hypothesis that NA tracks the emotional modulation of attention, we found that the NA response around stimulus presentation discriminated (1) between IAPS images that induced states of high versus low emotional arousal and (2) between the surprising IAPS images and the non-surprising checkerboard image within these two emotional states.

Comparison of the pupil and NA estimates supported the use of pupil dilation as a window into the LC-NA system.¹² In line with earlier reports that activity in LC neurons³³ and LC-NA axonal projections³⁴ can predict pupil dilation, we found a positive pupil-NA correlation for all trials regardless of stimulus type or emotional context. However, pupil dilation is influenced by multiple systems,¹² and there is evidence that pupil-NA coupling may change with cognitive state.^13,14 Indeed, we found a positive pupil-NA correlation for the surprising IAPS images in a high-arousal state but a negative correlation in a low-arousal state. While a negative correlation is unexpected, it can arise when two systems, such as the pupil and LC-NA systems,³⁵ are only driven by partially overlapping inputs. If there is not a strong common input to synchronize the systems, then the non-overlapping inputs may drive their responses, and if these inputs happen to go in the opposite directions, then negative correlations may arise.

Prior to this study, human electrochemistry has been performed during awake DBS surgery.^{15,16,17,18,19} With minimal deviations from the standard of care, and no reported changes in infection rates,³⁶ a custom-made carbon-fiber electrode can be inserted along the same path as the clinical electrodes used for functional mapping of the DBS target. The capacity to perform human electrochemistry in the EMU complements the DBS-based approach in multiple ways. It enables electrochemical recordings from a wider range of neural structures, and it allows for the study of physiological, behavioral, and cognitive processes that cannot be probed within the constraints of an acute surgical setting. For example, the LC-NA system is a pharmacological target in sleep disorders,³⁷ and it will be possible to monitor NA as patients transition between wakefulness and sleep. Similarly, a proportion of patients will be undergoing pharmacotherapy, such as selective serotonin reuptake inhibitors (SSRIs) or serotonin-NA reuptake inhibitors (SNRIs), for psychiatric symptoms, and it will be possible to compare neuromodulation before and after drug intake.

The brain’s major neuromodulatory systems support critical physiological, behavioral, and cognitive processes and have been implicated in a variety of clinical conditions. While high-precision methods for studying neuromodulation are available in animals,^38,39 similar advances are needed in humans to accelerate translation and advance our understanding of health and disease. The neuromodulatory machinery may be conserved across species, but it also operates on species-unique neural hardware and cognitive software in humans. By showing that neuromodulator estimates can be obtained from depth electrodes already in standard clinical use in the conscious human brain, our study opens the door to a new area of research on the neuromodulatory basis of human health and disease.

STAR★Methods

Key resources table

REAGENT or RESOURCE	SOURCE	IDENTIFIER
Chemicals, peptides, and recombinant proteins
Noradrenaline (NA)	Sigma-Aldrich	A7257; CAS: 51-41-2
Dopamine (DA)	Sigma-Aldrich	H8502; CAS: 62-31-7
Serotonin (5-HT)	Sigma-Aldrich	H9523; CAS: 153-98-0
Reagent for PBS: NaCl	Sigma-Aldrich	S7653-1K; CAS: 7647-14-5
Reagent for PBS: KCl	Sigma-Aldrich	P9333-1K; CAS: 447-40-7
Reagent for PBS: Na2HPO	Sigma-Aldrich	S7907-1K; CAS: 7558-79-4
Reagent for PBS: KH2PO4	Sigma-Aldrich	P5655-1K; CAS: 7778-77-0
Deposited data
Neural, pupil, and behavioral	This paper	https://github.com/danbang/article-NA-pupil-IAPS-oddball
Software and algorithms
pCLAMP	Molecular Devices	pCLAMP 10 Axon Instruments
MATLAB	MathWorks	MATLAB R2022a
Python	Van Rossum and Drake40	Python 3.9.7
R	R Core Team41	R 4.2.0
Stan	Carpenter et al.42	Stan 2.21.0
TensorFlow	Abadi et al.43	TensorFlow 2.6.0
Keras	Chollet44	Keras 2.6.0
Velocity-based blink detection algorithm with cubic-spline reconstruction	Mathôt and Vilotijević45	https://pydatamatrix.eu/
Code for reproducing figures	This paper	https://github.com/danbang/article-NA-pupil-IAPS-oddball
HMM tutorial	This paper	https://github.com/Beniamino92/mvHMM/tree/main/HMM-NE-pupil-IAPS-oddball
Other
Head stage	Molecular Devices	CV-7B-EC Axon Instruments
Amplifier	Molecular Devices	Multiclamp 700B Axon Instruments
A/D converter	Molecular Devices	Digidata 1440A Axon Instruments
Force function (TTL) generator	Tektronix	AFG320
Isolation transformer	Tripp Lite	IS500HG Isolation Transformer
Macro-micro electrode	Ad-Tech	MM16A-SP05X-000
Benhke-Fried electrode	Ad-Tech	MM16A-SP05X-000 (outer depth electrode) and WB09R-SP00X-0B6 (micro-wire bundle)
Eye-tracking system	Tobii	Tobii Pro Spectrum

Resource availability

Lead contact

Further information and requests for resources should be directed to and will be fulfilled by the lead contact, Dan Bang (danbang@cfin.au.dk).

Materials availability

This study did not generate new unique reagents.

Experimental model and study participant details

Patients

Six patients took part in the study; three patients performed the task (macro-micro; patient 1: female, 56 y; patient 2: female, 20 y; patient 3: male, 25 y) and three patients only contributed electrodes for in vitro evaluation (Behnke-Fried; patient 1: female, 62 y; patient 2: female, 27 y; patient 3: male, 28 y). The patients had medication-resistant epilepsy and undergone implantation of depth electrodes for localization of epileptic foci. Prior to the stay in the EMU, participation in the study was discussed with patients and the clinical team. The study protocol was described verbally and in a written format before patients provided verbal assent and written informed consent. No adverse or unanticipated events occurred during or as a result of the study protocol. The patient study was approved by the IRB committees at the Carilion Clinic (19-365) and Banner University Medical Center (STUDY00000295).

Controls

A cohort of seventeen adults (7 females, age range: 26-64 y) were recruited as controls (no reported history of psychiatric or neurological disorder) and performed the task in a standard laboratory setting. The control study was approved by the IRB committee at Virginia Tech (10-893).

Method details

Experimental task

Subjects performed a visual affective oddball task. The task was divided into six blocks of 100 images; each image was shown for 1 s and the images were separated by 1 s blank intervals. Within each block, 80% of the images were a checkerboard image (standard stimulus), whereas the remaining 20% were unique IAPS images (oddball stimuli). The presentation order was randomized within a block. The IAPS images were selected based on the image content ratings provided by the IAPS (Figure S2A) and grouped such that every block contained images of the same category: there were four emotionally evocative blocks, which varied in terms of valence (negative versus positive) and arousal (low versus high), and two emotionally neutral blocks. We included two neutral blocks to balance the design at the factor level (e.g., two neutral versus two high-arousal blocks). The order of the blocks was randomized within each subject. To keep subjects focused on the task, they were asked to press a button whenever an oddball stimulus was shown. The task was presented on a computer monitor mounted on top of the Tobii Pro Spectrum eye-tracking system. Subjects made a response by pressing any key on one of two button boxes, with a button box held in each hand.

Pupillometry

Data acquisition

The diameter of the pupil (pupil dilation) was monitored in the left and right eyes using the Tobii Pro Spectrum eye-tracking system running at a sampling rate of 1200 Hz. The subject was placed in a seated position 55-75 cm from the eye tracker as indicated by the Tobii Pro software. The system, which is fully external, was calibrated using a system-native 1 min protocol where the subject shifts their gaze to different corners of the computer screen.

Data pre-processing

The eye tracking system was unable to measure pupil diameter for some fraction of the samples taken, resulting in missing pupil diameter measurements. Sources of missing data include head movement, eyeblinks, and closing of the eyes. However, even when the eye tracking system is able to measure pupil diameter, the measurements may not always accurately reflect the actual pupil size, such as when the eyelid partly obscures the pupil. As is standard practice in pupillometry, we applied several pre-processing steps to reconstruct as much of the missing data as possible while correcting for as much of the invalid data as possible.

We adapted a velocity-based blink detection algorithm with cubic-spline reconstruction⁴⁵ for use with our specific eye-tracking system. In this algorithm, written in MATLAB, blinks are detected by the following sequence of features: (1) a high negative rate of change in pupil diameter (blink onset velocity); (2) a contiguous run of missing (or zero-valued) pupil diameter measurements (gap); (3) a high positive rate of change in pupil diameter (blink reversal velocity); and (4) a return to zero rate of change in pupil diameter (blink offset velocity). However, in our data, a large fraction of blinks did not exhibit the characteristic velocity excursions used by the original algorithm. To detect these blinks, we developed a second algorithm that detected gaps in pupil diameter measurements using the same gap duration thresholds as the velocity-based algorithm.

To reconstruct pupil diameter measurements across blinks, we also adapted the reconstruction itself. There was often significant noise in the pupil diameter measurements before and after blinks, likely induced by the eye-tracking system attempting to re-acquire the pupil. This issue required additional finesse in the reconstruction, resulting in these steps: (1) define a 5 s analysis window centered on the blink; (2) extract a smoothed (the "lowess" option to MATLAB’s "smooth" function, 0.125 s window) pupil diameter vector from the analysis window; (3) calculate mean and standard deviation of the smoothed pupil diameter values outside of the blink; (4) find four equidistant points in the smoothed vector within 1/2 of the blink duration before the blink onset; (5) find four equidistant points in the smoothed vector within 1/2 of the blink duration after the blink offset; (6) calculate the cubic-spline interpolation of the eight points; (7) replace the (unsmoothed) blink with the interpolated values; (8) reject any reconstruction which would result in velocity excursions exceeding the thresholds used for blink detection; and (9) reject any reconstructions which would result in pupil diameters exceeding three standard deviations from the mean smoothed pupil diameter (from step 3).

Since the various thresholding parameters for the blink detection and reconstruction in the original algorithm were reported in terms of samples (at a sampling rate of 1 kHz) and arbitrary units of pupil size, we had to convert the values to milliseconds and millimeters respectively before we could apply the algorithm to our eye-tracking data. We found that using fixed velocity thresholds did not perform well, so we set our thresholds in terms of the standard deviation of pupil velocity after smoothing (see Table S1 for a summary of parameters and conversion from original algorithm). After blink detection and reconstruction, we still noticed small gaps in the pupil data. Any such gaps lasting 0.08 s or less were corrected by applying a simple shape-preserving cubic spline interpolation (the "pchip" option to MATLAB’s "fillmissing" function). Before any data analysis, we averaged the reconstructed pupil data across the left and right eyes. If data was missing for one eye, then data from the other eye was used. Finally, the average pupil data was downsampled to 10 Hz using linear interpolation.

Data exclusion and quality

We excluded trials from pupil-related analyses when more than 50% of the data were reconstructed or missing following pre-processing (patient group: 24%; control group: 12%). For the included trials, 3% of the data were missing for a given time point in the patient group and 1% in the control group. While the proportion of excluded trials and missing time points is higher in the patient group, the presence of an oddball PDR in the patient group, and the correspondence between the oddball PDRs in the two groups, provide strong evidence for the reliability of the pupil data recorded in the EMU.

General description of electrochemical approach

Our approach builds on electrochemistry as applied in animals over the last three decades^46,47 and the recent adaptation of electrochemistry for use in the human brain.^15,16,17,18 However, the current study presents an advance in human electrochemistry, which to date has involved the insertion of research-exclusive carbon-fiber electrodes during awake DBS surgery, by showing that electrochemistry can be performed on depth electrodes that are already in standard clinical use. The data acquisition protocol in human electrochemistry is similar to fast-scan cyclic voltammetry (FSCV) as used in animals with regard to the time course of the voltage sweeps and the recording of the induced current time series during those sweeps.⁴⁸ The main change from animal work is the statistical method used to estimate the concentration of analytes of interest from the measured current time series.

FSCV involves the delivery of a rapid change in electrical potential to an electrode and measurement of the induced electrochemical reactions as changes in current at the electrode – with the guiding idea being that the current response carries information about both the identity and the concentration of analytes in the surrounding neural tissue. The goal of analysis of FSCV data is therefore to develop a statistical model that uses the current response in the best possible way to separate and estimate analytes of interest. The standard procedure is to train the statistical model on in vitro data collected in a laboratory setting where the presence and concentration of analytes of interest can be controlled and then apply this model to in vivo data for signal prediction.

Traditionally, the statistical model involves a decomposition of the in vitro training data into principal components that are then used for in vivo analyte inference within a regression framework.⁴⁹ In broad terms, this approach treats analyte inference as a problem of signal reconstruction: the concentration of an analyte of interest is estimated by mapping an in vivo current response onto those collected in vitro and then using the best match to label the in vivo current response. We instead treat analyte inference as a problem of signal prediction, with the statistical model optimized to generate accurate predictions about out-of-training data. Previous human work^16,17,18 has used elastic net regression,⁵⁰ but recent years have seen the development of more powerful machine learning methods. Here, we used deep convolutional neural networks as described below. Since information is distributed throughout a current time series and not only at the oxidation or reduction peaks typically revealed by principal components analysis,^16,17,18 we use non-decomposed data such that every time point within a current time series contributes to signal prediction. To facilitate out-of-training prediction, we train the model using large in vitro datasets and cross-validation as described below.

Earlier work has taken steps to validate this electrochemical approach. First, the human-compatible carbon-fiber electrodes have similar electrochemical properties to those used in the rodent brain.¹⁵ Second, the signal prediction approach returns more reliable neuromodulator estimates than principal component regression.¹⁶ Third, it does not confuse changes in pH for changes in neuromodulator levels.^16,17,18 Fourth, as shown here too (Figure S1), it can separate multiple neuromodulators from one another.^16,17,18,19 Fifth, it returns accurate neuromodulator estimates when tested in a laboratory setting where two neuromodulators change simultaneously across time.¹⁸ However, since these validations have been performed in vitro, future work should explore in vivo validation in animals; for example, one could evaluate signal predictions under optogenetically-controlled neuromodulator release³⁸ and compare the time course of these predictions to that measured using fiber photometry.³⁹

Implementation of electrochemistry in current study

Data acquisition

We conducted FSCV on standard Ad-Tech macro-micro electrodes (MM16A-SP05X-000) and Benhke-Fried electrodes (outer depth electrode: MM16A-SP05X-000; micro-wire bundle: WB09R-SP00X-0B6). The electrodes were stereotactically implanted into/explanted from the amygdala (macro-micro: left hemisphere in patient 1, right hemisphere in patients 2 and 3; Behnke-Fried: all left hemisphere). For the macro-micro electrodes, we used a longitudinally adjacent pair of micro-contacts on an electrode as our working and reference electrodes. For the Behnke-Fried electrodes, we used micro-wire #9 as our reference electrode and one of the other wires as our working electrode. Our FSCV protocol was a modification of previous work in both rodents^48,51 and humans.^{15,16,17,18,19} Our measurement waveform was a standard triangular voltage ramp applied at 10 Hz (macro-micro: hold at -0.6 V for 90 ms, ramp up from -0.6 V to +0.6 V at 240 V/s, ramp down from +0.6 V to -0.6 V at -240 V/s; Benhke-Fried: hold at -0.6 V for 90 ms, ramp up from -0.6 V to +0.175 V at 155 V/s, ramp down from +0.175 V to -0.6 V at -155 V/s) and we recorded the current response at 100 KHz. For in vivo experiments, we collected data for 2 min before the task to allow the electrode to equilibrate. We note that one of the macro-micro electrodes underwent changes in the current response during in vitro data collection. To maintain in vitro current traces similar to the in vivo ones, the triangular voltage ramp was decreased from ±240 V/s to ±160 V/s for all but the first in vitro dataset.

Signal prediction model

We generated in vivo signal predictions using an ensemble of deep convolutional neural networks that were trained and cross-validated on in vitro data with known concentrations of NA, DA, 5-HT and pH. The model architecture was based on the InceptionTime time series classification model⁵² but modified for a regression framework. The model was implemented in Python,⁴⁰ using TensorFlow⁴³ and Keras.⁴⁴ Following previous applications,⁵² equally weighted averages of in vivo signal predictions from multiple InceptionTime models were used to account for variability in the training process.

The (modified) InceptionTime model is based on two residual neural network (ResNet)⁵³ blocks, each containing three convolutional blocks. The input is added to the output of the first ResNet block, and, after activation, this serves as the input to the next ResNet block. It is also added to the output of the second ResNet block, and the sum flows through activation functions, global average pooling, and then finally to a dense layer with four output nodes, which after activation, gives the predictions of NA, DA, 5-HT, and pH. Each of the three convolutional layers in a ResNet block is further composed of convolutional blocks having four convolutional layers with 32 filters and increasing kernel sizes (1, 10, 20, and 40). The output of each of these convolutional layers is stacked together, after which batch normalization and activation are applied. This output serves as the input for the next convolutional block. All activation functions are RELU, except after the last dense layer, which uses softplus.

In vitro training data

We created specialized signal prediction models, by recovering the explanted electrode used for in vivo data collection and collecting in vitro data on this electrode for model training and evaluation. We collected four datasets. For each of the three neuromodulators (NA, DA, and 5-HT), a dataset was collected with 30 concentrations noisily dispersed over the range from 0 to 2500 nanomoles (nM) and with a pH of around 7.4. In addition, each of these datasets contained 5 mixture solutions in which the concentration of the neuromodulator in question was 840 or 1680 nM and one or two of the other neuromodulators had a concentration of 840 or 1680 nM. The fourth dataset was collected with 11 pH values in the range 7.0 to 7.8 and with the concentration of the three neuromodulators set to zero as well as 5 mixture solutions as described above. In each dataset, data collection was randomized across concentrations. For each concentration, the electrode was washed in PBS and then submerged in a PBS solution as described above. We applied our triangular voltage ramp at 10 Hz and recorded the current response at 100 KHz for 65 s. To reduce variation due to electrical noise and equilibration, we selected the most stable continuous 15 s section from the second half of the 65 s time window.

In vitro model training

For each electrode, a 10-fold cross-validation test was performed and then a signal prediction model was created. A training run involved a dataset being split into a training set containing around 90% of the data and a validation set containing the remainder. The data was split by concentration, such that all data points of any given concentration were in the same set. Before training, the training data were Z scored within analyte and then shifted by 10 standard deviations for each analyte to avoid zero gradients. After training, the predictions were then subjected to the inverse of the normalization procedure. A model was then trained on the training set using an ADAM optimizer,⁵⁴ with an initial learning rate of 1e-3, mean squared error loss and a batch size of 64. After each epoch, the loss on the validation set was calculated. If the performance on the validation set did not improve for five consecutive epochs, then the learning rate was halved until it reached a minimum of 1e-5. After 100 epochs were run, the model from the epoch with the lowest validation loss was selected as the final version. The training process was non-deterministic as variability is introduced by stochastic gradient descent, the initialization of the initial weights, and the order in which data is fed to the algorithm during training. When multiple training runs use the same data, different validation sets were used to reduce overfitting to any subset of the data.

In vitro model evaluation

To evaluate the sensitivity and specificity of the signal prediction models, a 10-fold cross-validation test was performed (Figures 1C and S1). The in vitro data was split into 10 discrete folds. Each fold was held out as a test set and an ensemble of four equally weighted sub-models was created via a training run on the remaining data. Ensembles of four models were used instead of five to reduce computation time. The mean of the predictions from each ensemble was calculated for its corresponding test set. All test sets were combined, resulting in all in vitro data points being predicted from ensembles that had never been trained or validated on data points with true concentrations of those being predicted.

In vivo signal prediction

In vivo signal predictions were generated using an ensemble of five equally weighted models from five training runs. No test set was held out to enable the model to train on the most data possible. In vivo current traces were submitted to each of the five models of the ensemble and the mean of the model predictions was used as the signal prediction.

Quantification and statistical analysis

Standardization

We standardized the data so that our group-level analyses could be performed in a relative frame of reference – where "large" and "small" values can be compared across patients. For example, for the analysis of the NA estimates shown in Figure 3A, we Z scored the trial-level NA estimates at the patient level, before running the linear mixed-effects regression at the group level. This approach minimizes, if not removes, the influence of unmodelled sources of patient-level variation in the baseline and/or the variance of the data.

Mixed-effects regression

All linear mixed-effects regression models consisted of fixed effects and a random intercept for each patient; model comparisons based on the Bayesian Information Criterion showed that the random-intercept models outperformed models which also included random effects. We applied the models at the trial level as this approach accommodates the difference in the number of standard and oddball trials. To illustrate our approach, we here consider the model for the estimated NA response around stimulus presentation. The model is specified as follows using Wilkinson notation:

$$NA\sim 1+(arousal\ast valence+evocative)\ast type+\left(1\right|subject)$$

In this model, stimulus type (standard = -1; oddball = 1) is specified at the trial level, and emotional valence (negative = -1; neutral = 0; positive = 1), emotional arousal (low = -1; neutral = 0; high = 1) and emotionally evocative (neutral = -1; evocative = 1) are specified at the block level as per our task design (Figure 2A). In addition, valence and arousal are considered features of blocks involving emotionally evocative IAPS images only (Figure S2A), with the evocative term capturing any difference between these blocks and blocks involving emotionally neutral IAPS images. We highlight that, because of our block design, a standard stimulus is not simply a non-IAPS image but a non-IAPS image of a particular category. This design feature is one of the reasons why it is possible to model interactions between stimulus type and the emotional variables.

Hidden Markov model

Hidden Markov models (HMMs), also known as Markov switching processes, are popular models for unsupervised statistical learning of multivariate time series data.^24,25 HMMs are especially suitable for data exhibiting non-stationary features that can be characterized by an underlying and unobserved hidden process. The approach assumes that the hidden process is in one of many latent states at any point in time, transitioning from one state to another over time, and that observations are generated from an emission distribution, conditional on the latent state sequence.

We implemented a bivariate HMM using R⁴¹ and Stan.⁴² The bivariate HMM was based on a discrete latent state sequence that partitions the pupil and NA estimates into different, potentially recurring regimes. Conditional on the latent state sequence, the two estimates are assumed to be generated from an emission distribution, here a bivariate Gaussian distribution with state-specific means and a state-specific variance-covariance matrix. The model allows a probabilistic estimate of the sequence of hidden states and the estimation of the state-specific parameters of the emission distribution.

Let $y_t=[NA\left(t\right),PD\left(t\right)]$ be the bivariate vector of the NA and pupil (PD) estimates at time point $t$, and let $z_t\in \{1,\dots ,K\}$ be the state of the hidden process at t, with $K$ denoting the overall number of latent states. The generative mechanism of our proposed model is:

$${p(y}_t|z_t=j)=N\left({\mu }_j,{\mathrm{\Sigma }}_j\right)$$

Where:

$${\mu }_j={[mean\left(NA\right),mean\left(PD\right)]}_j$$

$${\mathrm{\Sigma }}_j=\left[\begin{array}{cc}var\left(NA\right)& cov(NA,PD)\\ cov(PD,NA)& var\left(PD\right)\end{array}\right]\text{.}$$

We denote the state-specific transition probabilities as ${\pi }_j=[{\pi }_{j1},\dots ,{\pi }_{jK}]$, with ${\pi }_{ji}=p\left(z_t=i\right|z_{t-1}=j)$, according to the Markovian property that the probability of a change at time $t$ depends only upon the latent state at the previous time point $t-1$.

We adopted a Bayesian approach to inference. Within the Bayesian framework, the model parameters are regarded as random variables and inference is carried out via their posterior distribution, which by Bayes' rule is proportional to the likelihood of the data times the prior distribution.⁵⁵ In our model, the relevant parameters are ${\mu }_j$, ${\mathrm{\Sigma }}_j$ and ${\pi }_j$.

Our prior specification assumes a Dirichlet distribution on the transition probabilities ${\pi }_j$ and Student's t distributions on the mean parameters ${\mu }_j$. We further decompose the covariance matrix ${\mathrm{\Sigma }}_j$ into a vector of scale parameters ${\tau }_j$ and a correlation matrix ${\mathrm{\Omega }}_j$, and specify a half Student's t prior on ${\tau }_j$, which has been truncated to admit only positive values, and an LKJ distribution on the Cholesky factor of ${\mathrm{\Omega }}_j$. Our prior specification is summarized as follows. For each state $j=1,\dots ,J$

$$\begin{array}{c}{\pi }_j\sim \text{Dirichlet}\left(\mathrm{1,1,1}\right)\text{,}\\ {\mu }_{ji}\sim \text{Student's}t\left(4;0,1.5\right),i\in \left\{\text{NA},\text{PD}\right\}\\ {\tau }_{ji}\sim \text{Student's}{t}^{+}\left(4;1,1.5\right)\text{,}\\ L_{{\mathrm{\Omega }}_{\mathrm{j}}}\sim \text{LKJ}\left(1\right)\end{array}$$

where ${\pi }_j$ is the vector of transition probabilities, and ${\mu }_j=({\mu }_{jNA},{\mu }_{jPD})$ and ${\tau }_j=({\tau }_{jNA},{\tau }_{jPD})$ represent the vectors of state specific means and scales. The superscript ⁺ indicates that the distribution has been truncated to admit only positive values, and LKJ indicates the LKJ prior,⁵⁶ a family of probability distribution for positive definite correlation matrices (or equivalently for their Cholesky factors).

The posterior distribution is not available in closed form; we therefore used the No-U-Turn Sampler (NUTS) which leverages Hamiltonian dynamics to draw samples from the joint posterior distribution of the model parameters.

The number of latent states, $K$, for an HMM was selected by inspecting the posterior predictive fits and, in a more principled way, by calculating the ratio of marginal likelihoods from different models. The latter is often called the Bayes factor,⁵⁷ and can be thought of as the weight of evidence in favor of a model against a competing one. Bridge sampling⁵⁸ provides a general procedure for estimating marginal likelihoods in a reliable manner. This estimator can be implemented in R using the package bridgesampling,⁵⁹ whose compatibility with Stan makes it particularly straightforward to estimate the marginal likelihood directly from a Stan output.

Published: October 23, 2023

Data and code availability

• •De-identified neural, pupil, and behavioral data is available at GitHub (see key resources table).
• •Code for reproducing figures is available at GitHub (see key resources table).
• •Tutorial on HMM approach is available at GitHub (see key resources table).
• •Other data and code are available upon request.