Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2022 May 11.
Published in final edited form as: J Neural Eng. 2021 Nov 26;18(6):10.1088/1741-2552/ac3264. doi: 10.1088/1741-2552/ac3264

Brain-computer interface-based assessment of color vision

James J S Norton 1,2,*, Grace F DiRisio 3, Jonathan S Carp 1,4, Amanda E Norton 5, Nicholas S Kochan 6, Jonathan R Wolpaw 1,2
PMCID: PMC9094738  NIHMSID: NIHMS1792770  PMID: 34678801

Abstract

Objective:

Present methods for assessing color vision require the person’s active participation. Here we describe a brain-computer interface-based method for assessing color vision that does not require the person’s participation.

Approach:

This method uses steady-state visual evoked potentials to identify metamers—two light sources that have different spectral distributions but appear to the person to be the same color.

Main results:

We demonstrate that: minimization of the visual evoked potential elicited by two flickering light sources identifies the metamer; this approach can distinguish people with color-vision deficits from those with normal color vision; and this metamer-identification process can be automated.

Significance:

This new method has numerous potential clinical, scientific, and industrial applications.

Introduction

Accurate assessment of color vision is essential in many situations. Clinicians need it to detect and characterize the types and severities of the color vision deficits (CVDs) that affect >20% of the world population (1 ); industry needs it to develop and validate color schemes for physical products and digital displays; and scientists need it to study the neural mechanisms of color vision and its variations across people.

The anomaloscope—which is based on color matching—is the current gold-standard method for assessing color vision due to its ability to identify the type and severity of CVDs (2 ). With this method, the individual under study manually adjusts two light sources, one composed of a single wavelength and the other composed of two different wavelengths, until they appear to be the same color (i.e., are metamers; Fig. 1A). The light sources that different people perceive to be metamers vary due to differences in genetics, age, and sex (reviewed in (3 )). By analyzing metamers, it is possible to assess how individuals see colors and to detect and characterize CVDs.

Fig. 1:

Fig. 1:

Open in a new tab

Experiment 1. (A) Production of metameric stimuli. 525 nm (left) and 625 nm (middle-left) LEDs produce light that appears green and red, respectively. When appropriately combined (middle-right), these two LEDs form a dichromatic amber source that is a metamer of a monochromatic source from a 590 nm LED (right; also amber). (B) Experiment 1: Behavioral Session. Each of 8 participants with normal color vision adjusts the dichromatic source from an initially randomized setting (red square) to a setting (cyan circle) that produces a light metameric with a monochromatic source of 600 D/A units luminance. (C) Experiment 1: SSVEP session (Exp. 1B). SSVEP size is measured by canonical correlation analysis as a function of the luminance of the amber monochromatic source. Values are baselined and then averaged across runs and participants. The vertical dashed line indicates the amber monochromatic luminance of 600 D/A units which the participants matched in Exp. 1A. The three insets show for one participant the EEG power spectra obtained at three different luminances of the monochromatic source. The SSVEP fundamental (10-Hz) and second harmonic (20-Hz) peaks nearly disappear at a luminance of 600, which is when the monochromatic and dichromatic sources are metamers. (D) Spectrogram showing frequency-related changes in EEG activity for one participant as a function of the luminance of the amber monochromatic source. Data are the average of the three SSVEP runs for EEG channel Oz. As “C” and “D” illustrate, the SSVEP peaks are minimal or absent when the dichromatic and monochromatic lights are metamers.

Here, we present a new way to identify metamers. It is based on color matching, but—unlike existing methods—it does not require the active participation of the person being tested. Instead, it uses a non-invasive measure of brain activity (i.e., electroencephalography [EEG]) to quantify the brain’s response to flickering lights (i.e., the steady-state visual evoked potential [SSVEP]).

When a visual stimulus alternates between two light sources at a set frequency, chromaticity and/or brightness differences between the two sources elicit brain activity (i.e., an SSVEP) at the same frequency as the alternation (see (4 ) for review; Fig. S1).

We hypothesized that a stimulus alternating between metameric light sources will elicit little or no SSVEP. If this is correct, metamers should be identifiable as the pair of alternating light sources that minimize the SSVEP.

To test this hypothesis, we designed a stimulator that can produce metameric stimuli (Fig. S2). It comprises three independently-controlled LEDs with wave-lengths of 525 nm (green), 590 nm (amber), and 625 nm (red), respectively. These three LEDs can generate a monochromatic (amber) light source and a dichromatic (red and green) light source (Fig. 1A) that are metamers.

Results

In Experiment One, we compared behaviorally-identified metamers (Exp. 1A) with those identified using SSVEPs (Exp. 1B). We first asked participants (nine total participants, see Methods) to identify metamers by manually adjusting the dichromatic source of our stimulator to produce a color that matched the monochromatic light source, which had a fixed luminance setting of 600 digital-to-analog (D/A) units (see Methods; Fig. S3). This process established each person’s metameric pair. The mean ± SD luminance settings of the red and green components of the dichromatic source that this standard behavioral method found to be metameric with the monochromatic source were 135 ± 18 and 51 ± 6 D/A units, respectively (Fig. 1B). In Exp. 1B, we recorded EEG from the scalp and elicited SSVEPs by 10-Hz alternation between the two light sources. For each individual, we fixed the settings of the dichromatic source to be equal to those that Exp. 1A had found to be metamers and then adjusted the luminance of the monochromatic source from 0 to 1020 D/A units in steps of 20 (Figs. S4 and S5; see Methods). Across eight participants (data from one person were excluded, see Methods), SSVEP size was minimized when the monochromatic source had a setting of 575 ± 35 D/A units (Figs. 1C and 1D; topographical plots provided in Fig. S6 (5 )). This did not differ significantly from the setting of the monochromatic source found in the behavioral approach to metamer identification (i.e., 600 D/A units; one-sample Wilcoxon test, n=8, p=0.12). The SSVEPs elicited near this setting (540–720 D/A units) were significantly smaller than SSVEPs elicited at any other monochromatic source settings (one-way ANOVA; bins = 0–160, 180–340, 360–520, 540–700, 720–880, and 900–1020 D/A units; p< 0.001; Tukey post-hoc tests). Thus, the monochromatic source that was metameric with the dichromatic source in standard behavioral testing was also the source that minimized SSVEP amplitude.

In Experiment Two (nine participants, see Methods), we used a more extensive search of the workspace to ensure that the results of Exp. 1B were not merely a local minimum. With a two-dimensional grid search, we determined the red and green LED settings (i.e., the two dimensions) of the dichromatic source that minimized the SSVEP (Exp. 2B), and then compared them with the red and green LED settings that the person identified behaviorally (i.e., manually; Exp. 2A; Fig. 2A) as metamers of the same monochromatic source used in Experiment One. For each individual, we performed a coarse-grid search (stimulator settings of 0 to 500 D/A units in increments of 100 D/A units) and then a fine-grid search (red light settings of 75 to 200 D/A units in increments of 25 D/A units and green light settings of 25 to 75 D/A units in increments of 10 D/A units). In the coarse-grid search (data averaged across seven participants, see Methods), the SSVEP was minimized at 100 D/A units for both the red and green LEDs (Fig. 2B (left); analysis of grid data described in Fig. S7). These were the closest possible settings to the behaviorally-identified metamers. In the more focused fine-grid search (averaged data from all participants is shown in Fig. 2B (right)), the SSVEP was minimized when the green LED was 56 ± 6 D/A units and the red LED was 154 ± 22 D/A units. These green and red settings were very close to those of the behaviorally identified metamers (54 ± 6 for the green LED and 149 ± 16 D/A units for the red LED) (p=0.50 and 0.75, respectively, by Wilcoxon signed-rank test. Data from three individual participants are shown in Fig. 2C (top). Thus, the more complete sampling of the workspace in Exp. 2B confirmed the SSVEP results of Exp. 1B. In summary, the data of Experiments One and Two show that the SSVEP is minimized when the two stimuli are metameric; the SSVEP can identify metamers. The SSVEP-based method described here could be improved by methodological refinements in light sources, stimulation frequency and luminance, surrounding visual conditions, and EEG signal processing. Such improvements might further reduce the minimal SSVEP produced by the metameric pair (e.g., 1C).

Fig. 2:

Fig. 2:

Open in a new tab

(A) Experiment 2: Behavioral Session (Exp. 2A). Randomly initialized settings (red squares) and final settings (cyan circles) of the stimulator for 7 participants with normal color vision. The cyan circles are the settings of the dichromatic source that the participants identified as being metamers to the monochromatic source, which was at a fixed luminance of 600 D/A units. (B) Experiment 2: SSVEP Session (Exp. 2B). Results for the (left) coarse-grid search and (right) fine-grid search. The data from the two grid searches are the average of all five runs from all seven participants. (C) Experiment 3. Coarse-grid search results for: (top) three representative participants with normal color vision; and (bottom) the three participants with a CVD. Each person with normal color vision shows an SSVEP minimum that is close to zero and is focused in both the green and red dimensions; those with CVDs do not do so.

In Experiment Three, we investigated whether people with CVDs could be identified using our SSVEP-based method for finding metamers. People with CVDs see colors differently. Thus, we predicted that their metamers, as identified by SSVEP, would differ clearly from those of people without CVDs. To test this prediction we studied three people with CVDs (all male). All three were identified (by the Farnsworth Dichotomous D-15 arrangement test; see (6 )) as protans (i.e., they had a defective or missing L-cone). SSVEPs were elicited from these participants using the coarse-grid search of Exp. 2B (see Methods). Their results, shown in Fig. 2C (bottom), were markedly different from people without a CVD (Fig. 2C (top)) studied in Experiment Two. If these initial results are confirmed by studies in additional people, this new method could facilitate clinical detection of CVDs.

The SSVEP-based grid search is readily amenable to automated closed-loop operation. Thus, in Experiment Four, we tested a prototype SSVEP-based BCI that automatically identifies metamers. All of the tests conducted during Experiment Four were completed online (i.e., closed loop). As described in Methods, this system elicits and analyzes the person’s SSVEP in real-time while iteratively adjusting the luminances of the green and red LEDs that produce the dichromatic source until the SSVEP is minimized, thereby identifying a metamer to the monochromatic amber source (the luminance of which is fixed throughout). This process of identifying the specific settings of the dichromatic source that are metameric to the monochromatic source is a two-dimensional optimization problem, which we solved using gradient descent based on finite differences. Examples of the iterative changes made by the automated system are shown in Fig. 3.

Fig. 3:

Fig. 3:

Open in a new tab

Experiment 4: Automated BCI-based metamer identification in three people with normal color vision. Each panel shows for one person the initial (black) and final (red) stimulator settings for each of three automated runs. In each person, the three runs begin from different locations; nevertheless, they end up very close to each other. The data are overlaid on the average SSVEP results of Experiment 2 (i.e., Exp. 2B). As expected for these three people with normal color vision, the final locations reached by the automated search are at or very close to the average SSVEP minimumfor people with normal color vision.

Starting from three different initial settings, we tested the automated BCI-based system in three Experiment One participants (i.e., three individuals without CVDs; see Methods). Averaged across these three runs, the automated system identified 58 green, 117 red (S401); 76 green, 130 red (S402); and 56 green, 126 red (S403) as the settings that minimized the SSVEP (i.e., were metamers to the monochromatic source) for these three individuals. As illustrated in Fig. 3, each person’s three different runs arrived at very similar stimulator settings, despite starting from very different initial settings. Furthermore, a person’s metameric pair identified with the automated BCI-based system was very similar to that identified behaviorally in Experiment One, and by SSVEPs in Experiment Two. A simulation of how our BCI identified metamers (using data from S401) is shown in Supplemental Video 1. Future work might reduce the small differences between the BCI-identified metamers and the behaviorally-and SSVEP-identified metamers by optimizing the search parameters (e.g., step size, stopping criterion).

Discussion

In summary, Experiments One and Two show that, in people without CVDs, an SSVEP-based method can identify metameric pairs that match those identified by a standard behavioral method; Experiment Three shows that this SSVEP method can differentiate between people with and without CVDs; and Experiment Four shows that the method can be fully automated. The practical value of this new method for assessing color vision depends on its requirements and capabilities compared to current methods and on its potential for further development.

At present, color vision is evaluated mainly by behavioral methods, all of which require the attention and active participation of the person. Most prominent among these is the the anomaloscope, which is the only behavioral method able to diagnose both type and severity of red-green and blue-yellow color blindness (reviewed in (7 , 6 )). In addition to requiring the active participation of the person being examined, the anomaloscope requires extensive training of the examiner and considerable time to administer (8 ). In contrast, the SSVEP-based method described here does not require the person’s participation (i.e., it is a passive BCI (9 )); with appropriate signal analysis, it could even be used when the person’s eyes are closed (10 ). In addition, this new method needs minimal examiner training and can be fully automated. Furthermore, initial data (i.e., Exp. 3) indicate that it can detect protan-type CVDs accurately. (The new method can probably detect other types of CVDs as well, but this must be verified empirically.) Given these advantages SSVEP-based assessment of color vision could be particularly useful for identifying CVDs in those who cannot respond behaviorally, such as young children or people with motor or cognitive deficits.

The new SSVEP-based color-vision assessment method described here is fundamentally different from previous electrophysiological studies (11 , 12 , 13 , 14 , 15 ). It has four unique features. First, because it is based on the identification of metamers, it is an objective method for performing color matching, which is the gold standard of color-vision assessment. Second, by using new BCI-based procedures to analyze signals from multiple channels and frequencies, it enhances SSVEP detection and measurement (16 ). Third, because it poses metamer identification as an optimization problem, this new method can be fully automated and can take advantage of a wide range of powerful optimization algorithms. Fourth, the new method is generalizable; it can identify a metamer to a light source having essentially any spectral distribution.

The last two unique features – automatization and generalization – give the new method wide applicability for at least four important purposes. First, this new method could enable clinical detection and analysis of color vision deficits in young children and in others unable to participate in standard assessment methods. With further development, the method might be incorporated into a simple device that could be part of a standard pediatric examination. Second, the new method could find significant industrial applications in selecting color schemes for products and designing formats for digital displays (see (17 )).

Third, because SSVEPs reflect neural activity in cortical and subcortical areas involved in visual function (18 , 19 ), SSVEP-based detection of metamers could help to explore neural mechanisms underlying color vision (20 , 21 , 22 ). For example, it might make it possible to isolate and characterize the cortical responses to activation of intrinsically photosensitive retinal ganglion cells (ipRGCs), a photoreceptor type that contributes to visual function (reviewed in (23 )).

Fourth, this new method might form the basis for the first therapeutic intervention for people with CVDs. It is now clear that the simplest reflexes are plastic; they can be changed by operant conditioning, and these changes can help to restore useful function to people with spinal cord injury (24 ). Given that color vision displays comparable plasticity (25 ), an SSVEP-based operant conditioning protocol might prove able to improve color vision in people with CVDs.

Further studies should confirm the diagnostic ability of SSVEP-based BCIs to identify the type and severity of CVDs. As a part of these studies, improvements should be made to the stimulation system (e.g., optics, calibration) and protocol (e.g., identification of the optimal number and location of EEG electrodes, monitoring of the pupil size). Lastly, these studies should include direct comparisons to existing color vision assessment methods (e.g., the anomaloscope).

In conclusion, this study describes, demonstrates, and validates a novel method for assessing color vision founded on the hypothesis that flickering visual stimuli that alternate between two metamers will not elicit an SSVEP. Unlike standard color vision assessment methods, this SSVEP-based method does not need the active participation of the person being tested. The new method provides results comparable to those of standard methods, can identify those with color vision deficits (CVDs), can be fully automated, and can be applied to a wide variety of light sources. In addition to its clear clinical diagnostic applications, SSVEP-based testing of color vision should have industrial, scientific, and possibly therapeutic applications.

Materials and Methods

All experiments were approved by the institutional review board (IRB) of the New York State Department of Health’s Wadsworth Center.

Participants

Nineteen people were studied (five females and fourteen males, 20–72 years of age). Nine people completed Experiment One; nine people completed Experiment Two; three people completed Experiment Three; and three people completed Experiment Four. One person completed both Experiment One and Experiment Two (E1 (i.e., excluded) = S205), one person completed both Experiment Two and Experiment Four S201 = S401), and two people completed experiments one, two, and four (S107 = S206 = S402 and S105 = S207 = S403). The data from one participant was excluded from Experiment One (E1) due to a technical issue. Data from two participants were excluded from Experiment Two because they did not generate a measurable SSVEP. The color vision of each of the participants was screened using the first 25 plates of the 38 plate Ishihara pseudoisochromatic test (26 ) and/or the Farnsworth D-15 test (27 ). One of the participants without CVDs completed their screening online and two others self-reported having no CVD.

EEG Recording

EEG was recorded using a 16-channel g.USB (g.tec Medical Engineering GmbH, Austria) B-series amplifier. EEG electrodes were located in a mesh cap (Electro-Cap International, Inc., Eaton, OH) at the following 10–10 international system locations: F3, Fz, F4, T7, C3, CZ, C4, T8, CP3, CP4, P3, Pz, P4, PO7, PO8, and Oz (28 ). During the recordings, the EEG data were referenced to an electrode over the right mastoid and a ground electrode was placed on the left mastoid. Conductive gel was used to reduce to impedance to <40 kΩ. All data were acquired using BCI2000 (29 ) at a sampling rate of 256 Hz. No bandpass or notch filters were used during acquisition.

Stimulation System

A custom-built stimulation system, consisting of a stimulator, microcontroller, and software interface was used to elicit SSVEPs. Diagrams of the stimulator (including the emitter) and BCI system are shown in Fig. S2. The stimulator consisted of a four-die (red [~625 nm; full width at half maximum (FWHM) = 20 nm], amber [~590 nm; FWHM = 20 nm], green [~525 nm; FWHM = 35 nm], and blue [~465 nm]) LED emitter (LZA4–00MA00; LED Engin, Inc., San Jose, CA), heat sink, constant current LED driver (DD313, Silicon Touch Technology, Inc., Hsin-Chu, Taiwan), and 3D printed housing. In this paper, we refer to each die as an LED.

The stimulator was controlled by a Teensy microcontroller (PJRC, Sherwood, OR) running Arduino (Arduino LLC, Somerville, MA). A software interface was developed with MATLAB (The Mathworks Inc., Natick, MA) to allow the experimenters to digitally adjust the luminance of each LED using pulse-width modulation (PWM; see Fig. S5).

PWM adjusts the proportion of the time that the voltage input to the LED is high (i.e., on). In our design, PWM was adjusted with 10-bits of precision (i.e., there were 1024 possible luminance settings for each LED). We refer to these potential settings as D/A units. A setting of 0 denoted that the input to the LED was high 0% of the time and a setting of 1023 represented that the input voltage to the LED was high 100% of the time.

Digital control of the luminance of the LEDs starts at the computer (Figs. S2 and S5). A command to increase the luminance of one of the LEDs is sent from the computer to the Teensy microcontroller via USB. The computer command causes the Teensy microcontroller to increase the duty cycle (in D/A units) of an analog output pin using PWM. PWM adjusts the proportion of the time that the voltage output of an analog output pin is high. In our design, PWM was adjusted with 10-bits of precision (i.e., there were 1024 possible settings; measured in D/A units from 0 to 1023). This increased duty cycle from the analog output pin is then transmitted to the constant current controller. The constant current controller adjusts the current to the LED based on PWM (i.e., higher values of PWM output higher current). Given the fixed position of the LED and diffusers, the projected geometry of the light passing through the stimulator is also fixed. Thus, increasing current increases luminance as perceived by the participant.

Procedure

All experiments were conducted at the David Axelrod Institute, Wadsworth Center, Albany, NY in a room with consistent ambient lighting (~300 lux).

After completing the informed consent process, participants were asked to sit in a comfortable desk chair for the duration of the study. A chin rest was used during all of the sessions to stabilize the position of each participant relative to the stimulator. Participants were about one foot from the stimulator; it subtended a visual angle of ~10°.

Experiment One

Experiment One compared behaviorally-identified metamers with those identified using SSVEPs.

Behavioral Session

To account for individual differences in color (30 , 3 ), a behavioral session (Exp. 1A) was used to find a combination of red and green lights that had the same color as the amber light at a predetermined luminance setting (600 D/A units). This behavioral session was completed in multiple phases (Fig. S3): initialization; iterative adjustment process (i.e., single-and dual-LED calibration); and stopping.

Initialization -

During initialization, the amber light (monochromatic source) was always set to have an D/A setting of 600. The red and green lights (dichromatic source), however, were randomly set to have D/A settings between 0 and 255. After initialization, the participant was asked for guidance on how to adjust the stimulator (e.g., is the test source too green?). Based on this guidance, the experimenter could choose one of two different actions. For example, if the participant indicated that the test source was too green, the experimenter could either (1) decrease the luminance of the green LED or (2) increase the luminance of the red LED. The choice of action was left to the discretion of the experimenter.

Single-LED calibration -

A two-interval discrimination task (2IDT) was used to adjust the dichromatic source to have the same color as the monochromatic source (Fig. S3(b)). To do this, the participants were asked to judge which of two test sources was closer in color to the monochromatic source. The two alternatives consisted of different combinations of light from the red and green lights. The luminances of the red and green lights were adjusted separately (red first and then green). As an example of adjusting the red light, two alternative dichromatic light combinations are presented to the participant. In Alternative 1, the red and green lights have D/A settings of 250 and 50 respectively, while in Alternative 2, the red and green lights have D/A settings of 200 and 50 respectively. During this presentation, the stimulator switches between the monochromatic source and one of the two alternative dichromatic sources at a rate of 1 Hz. Each presentation lasts for three seconds. The participant is then asked which of the two alternatives is more similar in color to the monochromatic source. If the participant is unsure, they are permitted to see each alternative again. Based on the participant’s answer, the luminance of the red light is then increased or decreased. The relative size of the luminance change—known as the increment —was determined a priori and started at 50 D/A units. Continuing this example, if the participant chooses Alternative 1, the new 2IDT would be between Alternative 1’s current red and green D/A settings and a new alternative (i.e., Alternative 3) with D/A settings of 300 and 50 If the participant chooses Alternative 3, the process continues. If the participant chooses Alternative 1 again, the size of the increment is reduced (from 50 to 25 after the choice is repeated once, and then from 25 to 10 D/A units after the choice is repeated a second time). The process is then resumed with a new alternative (i.e., Alternative 4, with settings of 275 and 50]). Calibration of the red light is stopped when the increment equaled 10 and the same alternative is chosen twice in a row. This process is then repeated for the green light.

Dual-LED calibration –

Dual-LED calibration (where both the red and green LEDs were adjusted simultaneously) was based on heterochromatic flicker photometry (HFP) (31 , 32 , 33 ). Originally described by Walsh (31 ), HFP is a method for comparing the brightness of two light sources(32 ). When flickered sufficiently fast (i.e., above the critical flicker fusion rate (31 )) light sources of equal brightness have minimal perceived flicker. In our study, participants were asked to minimize the flicker between two light sources at a single location using a 2IDT (Fig. S3 (c)). Each of the two alternatives consisted of the monochromatic source and a dichromatic source. By alternating between these two light sources (at 25Hz)(34 , 35 ), a flickering stimulus was generated. As an example, Alternative 1 consisted of the monochromatic source and a dichromatic source with red and green D/A settings of 200 and 50. Alternative 2 consisted of the monochromatic source and a dichromatic source with red and green D/A settings of 100 and 25. These two alternatives were then presented to the participant for three seconds each. The participant was then asked which of the two alternatives flickered less. If the participant was unsure, they were permitted to see each alternative again. Based on the participant’s answer, the luminance of the red and green lights was then increased or decreased. To primarily adjust the brightness of the test source, the luminance of both the red and green lights were increased or decreased simultaneously. As in the monochromatic calibration, the amount that the lights were adjusted after each decision was defined as the increment. The increment during dichromatic calibration started at 100 D/A units. Continuing our example, if the participant chose Alternative 2, the new 2IDT would be Alternative 2 and a new alternative (i.e., Alternative 3) with increased D/A settings (300 and 75). If the participant chose Alternative 3, the process continued. If the participant chose Alternative 2 again, the size of the increment was reduced (from 100 to 50 after the choice is repeated once, then from 25 to 10 D/A units after the choice is repeated a second time) and a new alternative was presented. Dichromatic calibration stopped when the increment equaled 10 and the same alternative was chosen twice in a row.

Stopping criteria –

Both the single-LED and dual-LED calibrations were each repeated until three stopping criteria were met. First, both the single-LED and dual-LED calibrations were completed at least once. Second, the total difference between the current red and green light D/A settings were no more than 10 units different from the red or green light D/A settings from the previous single-LED or dual-LED calibration. Third, the participant confirmed that the monochromatic source and dichromatic source appeared to be equivalent. If all of these stopping criteria were met, then the adjustment was considered complete and the red and green settings determined during the behavioral session were saved for use during the SSVEP session. If the stopping criteria were not met and single-LED calibration was just completed, then dual-LED calibration proceeded using the current red and green D/A settings. If the stopping criteria were not met and dual-LED calibration was just completed, then single-LED calibration proceeded using the current red and green D/A settings.

SSVEP Session

After the behavioral session, participants completed an SSVEP session (Exp. 1B). For three of the participants, both sessions were completed on the same day. The other six participants completed the SSVEP session on a different day (within one week). During the SSVEP session, EEG was recorded from the participants as described in Methods: EEG Recording. Following setup, each participant was given the opportunity to make final adjustments to the settings of the stimulator. After this, participants completed three runs of stimulation.

Each run of stimulation consisted of 54 six-second trials with an interstimulus interval (ISI) of one second. Previous research has shown that canonical correlation analysis (CCA) can classify SSVEPs with >75% accuracy in 2.25 s (36 ). Our goal was to estimate the size of SSVEPs, which we thought may require more data than classification. Thus, we chose to make each trial 6 s. The onset of each of these trials was detected using the digital input line of the g.USB amplifier. The order of the trials was the same in every run for every participant. Randomizing the order of the trials could have resulted in large relative luminance changes from trial to trial (see Fig. S5). Large changes in luminance could lead to large changes in pupil dilation (37 ), potentially introducing noise into the experiments. The first two trials of each run measured baseline levels of EEG activity. In the first baseline trial, participants were asked to attend to the stimulator while it was off. In the second baseline trial, the monochromatic source was turned on for the entire trial (i.e., there was no flicker). During each of the remaining 52 trials, the stimulator flickered at 10 Hz. This flicker was obtained using square wave stimulation (with a 50% duty cycle) that switched between the monochromatic source and the dichromatic source. The settings of the dichromatic source were fixed throughout the SSVEP session. The luminance of the monochromatic source, however, was increased from a D/A setting of 0 in the third trial by 20 D/A units each trial to 1020 D/A units in the last trial. Each run of stimulation lasted for less than seven minutes. Participants were allowed to rest for approximately five minutes between runs. The entire SSVEP session lasted less than one hour.

Experiment Two

There were two key differences between Experiment One and Experiment Two. First, during Experiment Two, the SSVEP session came before the behavioral session. Second, during the SSVEP session of Experiment Two, only the monochromatic source was fixed. The method used during the behavioral session of Experiment Two (Exp. 2A) was identical to Experiment One.

SSVEP Session (grid search)

In Experiment Two: SSVEP Session (grid search; Exp. 2B), the D/A settings of the red light and the green light that each person perceived as being metameric with the monochromatic source were assumed to be unknown. Therefore, the settings of the red light and green light that minimized the SSVEP were determined using two grid searches Fig. S7), a coarse-grid search followed by a fine-grid search. In both searches, 36 possible combinations of red and green light settings were tested. In the coarse-grid search, green light settings of 0 to 500 D/A units in increments of 100 D/A units and red light settings of 0 to 500 D/A units in increments of 100 D/A units were tested. In the fine grid search, red light settings of 75 to 200 D/A units in increments of 25 D/A units and green light settings of 25 to 75 D/A units in increments of 10 D/A units were tested. These combinations were tested in order of increasing D/A setting (i.e., the trials were ordered from the lowest sum of the D/A settings to the highest). The D/A settings tested during the fine-grid search were chosen based on the results of the behavioral session of Experiment One.

The length of each trial, ISI, and EEG recording parameters of Experiment Two: SSVEP Session were identical to those used in Experiment One: SSVEP Session.

Experiment Three

The procedure for Experiment Three was identical to that of Experiment Two: SSVEP Session, except that the participants did not complete the fine-grid search.

Experiment Four

Experiment Four used the same EEG and Data Analysis settings as the other experiments. As opposed to conducting a grid search, however, the setting that minimized the SSVEP was identified using gradient descent based on finite differences. The stimulator was initialized to three settings chosen by the experimenter. For S401, these settings were 150 green and 150 red; 15 green and 30 red; and 0 green and 100 red. For S402, these settings were 0 green and 0 red; 0 green and 100 red; and 100 green and 0 red. For S403, these settings were 40 green and 50 red; 30 green and 100 red; and 100 green and 40 red. The system then sampled the settings surrounding the initial setting, computed a gradient, and updated its current estimate of the settings that minimized the SSVEP. This process was then repeated and continued until the difference between the different settings surrounding the current estimate was sufficiently small (i.e., a stopping criterion was met).

Data Analysis

Data analysis for all four experiments was performed using Matlab. The raw EEG data from each participant was zero-phase bandpass filtered using a 4th order IIR Butterworth filter between 3 and 45 Hz. The data was also notch filtered at 60 Hz to remove powerline noise. After filtering, individual six-second trials were extracted from the EEG data. We analyzed each trial using canonical correlation analysis (CCA) (36 ). CCA is a technique—widely used in BCI—for detecting SSVEPs. Here we use CCA as a relative measure of the size of the SSVEP elicited during each trial (assuming that the noise in the EEG is stable across trials). Details on our use of CCA for detecting SSVEPs was based on those described by Norton et al. (38 ). Our CCA analysis included all 16 channels of EEG data and reference variables (sine and cosine waves) at the first, second, third, fourth, and fifth harmonic frequencies (i.e., 10, 20, 30, 40, and 50 Hz) of the stimulation. Although CCA calculates multiple canonical correlations (the number of canonical correlates is the lesser of the number of reference variables and the number of EEG channels), we discarded all but the maximum canonical correlation in our analysis. After performing CCA on each of the trials, the data from each run were normalized between zero and one.

Supplementary Material

Supplemental Video
Download video file (11.6MB, mp4)
Supplemental Figures

Acknowledgements

The authors would like to thank everyone at the National Center for Adaptive Neurotechnologies who have provided feedback and advice on this project, including Theresa Vaughan, Dennis McFarland, William (Billy) Schmidt, and Olivia Zhou. We would also like to thank Andy Borum for his helpful discussions on optimization, Scott Carney for his recommendations on improvements to the stimulator, and Tim Bretl and Aadeel Akthar for their advice. Thank you to Zeyuan Yu for his help building the stimulator. Finally, thank you to Susan Heckman, Penelope Norton, and Stephanie Dockins for their help in improving the manuscript.

Funding:

The National Center for Adaptive Neurotechnologies is supported by the National Institute of Biomedical Imaging and Bioengineering of the NIH (Grant P41 EB018783–06 (JRW)). Work in the authors’ laboratory has also been supported by NIH grants R01 EB026439–03 (Peter Brunner), U24 NS109103–02 (Peter Brunner), R01 NS110577–01A1 (JRW, JC, Yu Wang), R25 HD088157–03 (JRW), by VA Merit Award 5I01CX001812–02 (JRW), and by the New York State Spinal Cord Injury Research Board (SCIRB) grants C32236GG (JRW) and DOH01-C33279GG-3450000 (JRW). This material is the result of work supported with resources at the Stratton VA Medical Center in Albany, NY.

References

  • [1].Delpero Walter T., O’Neill Hugh, Evanne Casson, and Hovis Jeff. Aviation-elevent epidemiology of color vision deficiency. Aviation, Space, and Environmental Medicine, 76(2):127–133, 2005. [PubMed] [Google Scholar]
  • [2].Squire Theresa J., Rodriguez-Carmona Marisa, Evans Anthony D. B., and Barbur John L.. Color vision tests for aviation: Comparison of the anomaloscope and three lantern types. Aviation, Space, and Environmental Medicine, 76(5):421–429, 2005. [PubMed] [Google Scholar]
  • [3].Webster Michael A.. Individual differences in color vision. In Elliot Andrew J., Fairchild Mark D., and Franklin Anna, editors, Handbook of Color Psychology, pages 197–215. Cambridge University Press, Cambridge, 2015. doi: 10.1017/cbo9781107337930.010. [DOI] [Google Scholar]
  • [4].Vialatte François-Benoît, Maurice Monique, Dauwels Justin, and Cichocki Andrzej. Steady-state visually evoked potentials: Focus on essential paradigms and future perspectives. Progress in Neurobiology, 90(4):418–438, Apr 2010. doi: 10.1016/j.pneurobio.2009.11.005. [DOI] [PubMed] [Google Scholar]
  • [5].Delorme Arnaud and Makeig Scott. EEGLAB: An open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. Journal of Neuroscience Methods, 134(1):9–21, 2004. [DOI] [PubMed] [Google Scholar]
  • [6].Fanlo Zarazaga A, Gutiérrez Vásquez J, and Pueyo Royo V. Review of the main colour vision clinical assessment tests. Archivos de la Sociedad Española de Oftalmologίa (English Edition), 94(1):25–32, jan 2019. doi: 10.1016/j.oftale.2018.08.010. [DOI] [PubMed] [Google Scholar]
  • [7].Dain Stephen J.. Clinical colour vision tests. Clinical and Experimental Optometry, 87(4–5):276–293, jul 2004. doi: 10.1111/j.1444-0938.2004.tb05057.x. [DOI] [PubMed] [Google Scholar]
  • [8].Walsh David V., Robinson James, Jurek Gina M., Capó-Aponte José E., Riggs Daniel W., and Temme Leonard A.. A performance comparison of color vision tests for military screening. Aerospace Medicine and Human Performance, 87(4):382–387, 2016. [DOI] [PubMed] [Google Scholar]
  • [9].Aricò P, Borghini G, Di Flumeri G, Sciaraffa N, and Babiloni F. Passive BCI beyond the lab: Current trends and future directions. Physiological Measurement, 39(8):08TR02, 2018. [DOI] [PubMed] [Google Scholar]
  • [10].Norton James J. S., Umunna Stephen, and Bretl Timothy. The elicitation of steady-state visual evoked potentials during sleep. Psychophysiology, 54 (4):496–507, 2017. [DOI] [PubMed] [Google Scholar]
  • [11].Allen Dale, Banks Martin S., and Norcia Anthony M.. Does chromatic sensitivity develop more slowly than luminance sensitivity? Vision Research, 33:2553–2553, 1993. [DOI] [PubMed] [Google Scholar]
  • [12].Gerling Jürgen, Meigen Thomas, and Bach Michael. Diagnosis of protan and deutan color vision deficiencies with pattern-ERG and VEP. In Colour Vision Deficiencies XII, pages 375–380. Springer, 1995. [Google Scholar]
  • [13].Ver Hoeve James N., France Thomas D., and Andrew Bousch G. A sweep VEP test for color vision deficits in infants and young children. Journal of Pediatric Ophthalmology and Strabismus, 33(6):298–302, 1996. [DOI] [PubMed] [Google Scholar]
  • [14].Gomes Bruno D., Souza Givago S., Rodrigues Anderson R., Saito Cezar A., Silveira Luiz Carlos L., and Da Silva Filho Manoel. Normal and dichromatic color discrimination measured with transient visual evoked potential. Visual Neuroscience, 23(3–4):617–627, 2006. [DOI] [PubMed] [Google Scholar]
  • [15].Rabin Jeff C., Kryder Andrew C., and Lam Dan. Diagnosis of normal and abnormal color vision with cone-specific veps. Translational Vision Science & Technology, 5(3):8–8, 2016. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [16].Nan Wenya, Wong Chi Man, Wang Boyu, Wan Feng, Mak Peng Un, Mak Pui In, and Vai Mang I.. A comparison of minimum energy combination and canonical correlation analysis for ssvep detection In 2011 5th International IEEE/EMBS Conference on Neural Engineering, pages 469–472. IEEE, 2011. [Google Scholar]
  • [17].Hunt R. The reproduction of colour sixth edition John Wiley & Sons, 2004. [Google Scholar]
  • [18].Russo Francesco Di, Pitzalis Sabrina, Aprile Teresa, Spitoni Grazia, Patria Fabiana, Stella Alessandra, Spinelli Donatella, and Hillyard Steven A.. Spatiotemporal analysis of the cortical sources of the steady-state visual evoked potential. Human Brain Mapping, 28(4):323–334, 2007. doi: 10.1002/hbm.20276. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [19].Pastor María A., Valencia M, Artieda Julio, Alegre Manuel, and Masdeu Jose C.. Topography of cortical activation differs for fundamental and harmonic frequencies of the steady-state visual-evoked responses. An EEG and PET H215O study. Cerebral Cortex, 17(8):1899–1905, Oct 2007. doi: 10.1093/cercor/bhl098. [DOI] [PubMed] [Google Scholar]
  • [20].De Valois Russell L. and De Valois Karen K.. A multi-stage color model. Vision Research, 33(8):1053–1065, 1993. [DOI] [PubMed] [Google Scholar]
  • [21].Mancuso Katherine, Mauck Matthew C., Kuchenbecker James A., Neitz Maureen, and Neitz Jay. A multi-stage color model revisited: Implications for a gene therapy cure for red-green colorblindness. In Retinal Degenerative Diseases, pages 631–638. Springer, New York, 2010. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [22].Schmidt Brian P., Neitz Maureen, and Neitz Jay. Neurobiological hypothesis of color appearance and hue perception. Journal of the Optical Society of America A, 31(4):A195–A207, 2014. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [23].Schmidt Tiffany M., Chen Shih-Kuo, and Hattar Samer. Intrinsically photosensitive retinal ganglion cells: Many subtypes, diverse functions. Trends in Neurosciences, 34(11):572–580, 2011. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [24].Norton James J. S. and Wolpaw Jonathan R. Acquisition, maintenance, and therapeutic use of a simple motor skill. Current Opinion in Behavioral Sciences, 20:138–144, 2018. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [25].Neitz Jay, Carroll Joseph, Yamauchi Yasuki, Neitz Maureen, and Williams David R.. Color perception is mediated by a plastic neural mechanism that is adjustable in adults. Neuron, 35(4):783–792, aug 2002. doi: 10.1016/s0896-6273(02)00818-8. [DOI] [PubMed] [Google Scholar]
  • [26].Ishihara Shinobu. Tests for color blindness. American Journal of Ophthalmology, 1(5):376, may 1918. doi: 10.1016/s0002-9394(18)90663-x. [DOI] [Google Scholar]
  • [27].Farnsworth Dean. Farnsworth dichotomous test for color blindness 1947. [Google Scholar]
  • [28].Chatrian GE, Lettich E, and Nelson PL. Ten percent electrode system for topographic studies of spontaneous and evoked EEG activities. American Journal of EEG Technology, 25(2):83–92, jun 1985. doi: 10.1080/00029238.1985.11080163. [DOI] [Google Scholar]
  • [29].Schalk G, McFarland DJ, Hinterberger T, Birbaumer N, and Wolpaw JR. BCI2000: A general-purpose brain-computer interface (BCI) system. IEEE Transactions on Biomedical Engineering, 51(6):1034–1043, jun 2004. doi: 10.1109/tbme.2004.827072. [DOI] [PubMed] [Google Scholar]
  • [30].Kuehni Rolf G.. Variability in estimation of suprathreshold small color differences. Color Research & Application, 34(5):367–374, oct 2009. doi: 10.1002/col.20522. [DOI] [Google Scholar]
  • [31].Walsh John W. T.. Modern photometry. Nature, 120(3013):157–157, jul 1927. doi: 10.1038/120157a0. [DOI] [Google Scholar]
  • [32].Bone Richard A. and Landrum John T.. Heterochromatic flicker photometry. Archives of Biochemistry and Biophysics, 430(2):137–142, oct 2004. doi: 10.1016/j.abb.2004.04.003. [DOI] [PubMed] [Google Scholar]
  • [33].Teikari Petteri, Raymond P Najjar Kenneth Knoblauch, Dumortier Dominique, Cornut Pierre-Loïc, Denis Philippe, Cooper Howard M, and Gronfier Claude. Refined flicker photometry technique to measure ocular lens density. JOSA A, 29(11):2469–2478, 2012. [DOI] [PubMed] [Google Scholar]
  • [34].Hung Shao-Min, Milea Dan, Venkata Rukmini Annadata, Najjar Raymond P., Tan Joo Huang, Viénot Françoise, Dubail Marie, Choon Tow Sharon Lee, Aung Tin, Gooley Joshua J., and Hsieh Po-Jang. Cerebral neural correlates of differential melanopic photic stimulation in humans. NeuroImage, 146:763–769, feb 2017. doi: 10.1016/j.neuroimage.2016.09.061. [DOI] [PubMed] [Google Scholar]
  • [35].Brown Timothy M., Tsujimura Sei ichi, Allen Annette E., Wynne Jonathan, Bedford Robert, Vickery Graham, Vugler Anthony, and Lucas Robert J.. Melanopsin-based brightness discrimination in mice and humans. Current Biology, 22(12):1134–1141, jun 2012. doi: 10.1016/j.cub.2012.04.039. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [36].Lin Zhonglin, Zhang Changshui, Wu Wei, and Gao Xiaorong. Frequency recognition based on canonical correlation analysis for SSVEP-based BCIs. IEEE Transactions on Biomedical Engineering, 54(6):1172–1176, jun 2007. doi: 10.1109/tbme.2006.889197. [DOI] [PubMed] [Google Scholar]
  • [37].Watson Andrew B and Yellott John I. A unified formula for light-adapted pupil size. Journal of Vision, 12(10):12–12, 2012. [DOI] [PubMed] [Google Scholar]
  • [38].Norton James J. S., Mullins Jessica, Alitz Birgit E., and Bretl Timothy. The performance of 9–11-year-old children using an SSVEP-based BCI for target selection. Journal of Neural Engineering, 15(5):056012, jul 2018. doi: 10.1088/1741-2552/aacfdd. [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplemental Video
Download video file (11.6MB, mp4)
Supplemental Figures
NIHMS1792770-supplement-Supplemental_Figures.pdf (315.4KB, pdf)

RESOURCES