InFoRM (Indicate-Follow-Replay-Me): a novel method to measure perceptual multistability dynamics using continuous data tracking and validated estimates of visual introspection

Abstract

Perceptual multistability, e.g. Binocular Rivalry, has been intensively used as a tool to study visual consciousness. Current methods to assess multistability do not capture all potentially occurring perceptual states, provide no estimate of introspection, and lack continuous, high-temporal resolution to resolve perceptual changes between states and within mixed perceptual states. We introduce InFoRM (Indicate-Follow-Replay-Me), a four-phase method that (1) trains a participant to self-generate estimates of perceptual introspection-maps that are (2) validated during a physical mimic task, (3) gathers perceptual multistability data, and (4) confirms their validity during a physical replay. 28 condition-blinded adults performed InFoRM while experiencing binocular rivalry evoked with orthogonal sinusoidal gratings. A 60Hz joystick (3600data samples/minute) was used to indicated continuously changes in perception of six perceptual states within each 1min trial. A polarized monitor system was used to present the stimuli dichoptically. Three contrast conditions were investigated:. low vs. low, high vs. high, and low vs. high. InFoRM replicates standard outcome measures, i.e. alternation rate, mean and relative proportions of perception, and distribution of exclusive percepts that is well fitted with a gamma function. Furthermore, InFoRM generates novel outcomes that deliver new insights in visual cognition via estimates of introspection maps, in ocular dominance via perceptual-state-specific dominance scores, in transitory dynamics between and within perceptual states, via techniques adopted from eye-tracking, and in rivalry-zone-size estimates utilizing InFoRM’s ability to simulate piecemeal perception. The replay phase (physical replay of perceptual rivalry) confirmed good overall agreement (73% ±5 standard deviation). InFoRM can be applied to other multistable paradigms and can be used to study visual consciousness in typical and neuro-atypical populations.

1. Introduction

Neuro- and behavioral sciences aim to gain insights about perceptual experiences of conscious creatures, in particular humans. The difficulty of measuring the nature of subjective experiences has been famously described by the philosopher Thomas Nagel in his famous essay “What is it like to be a bat?”, in which he considers the difficulty of gaining insights into the subjective experiences of another organism (Nagel, 1974). Later, the study of consciousness was subdivided into the “easy problem” i.e. study of the mechanisms of perception and action, and the “hard problem”, i.e. the study of the experience surrounding any conscious perception and action (Chalmers, 1996). While qualitative research aims to document individual conscious experiences by means of e.g. interviews or questionnaires; quantitative approaches use either report- or no-report paradigms to infer similarities and difference in perception and action between individuals and groups. In the case of report paradigms, the participant is asked to carry out a specific task and an additional action to consciously report perception and changes thereof. One very popular report paradigm, in use for over 100 years (Wheatstone, 1838), is binocular rivalry (Alais, 2012; Alais & Blake, 2005). For examples, during a set of multistability experiments by Einhäuser, Stout, Koch, & Carter (2008), the participants were asked to press one of two buttons on a keyboard as soon as perception alternated between exclusive percepts. In the case of no-report paradigms no action is required during the perceptual experience of multistability (Alais, Keetels, & Freeman, 2014), instead physiological measures are used to infer perceptual alternation and correlated with previously-collected report-based outcomes. For example, the Einhäuser and colleagues study described above also measured pupil size changes and showed that pupils constricted shortly before and dilatated shortly after a perceptual alternation, allowing them to use pupil size changes as robust predictor of a perceptual change without asking participants to report their perceptual experiences directly. It is noteworthy that report-based findings are still required to establish the physiological-perceptual relationship and further that the participants in that study only indicated perception of two exclusive states. We will elaborate on this point further below.

Within the domain of sciences of the mind, one method, known as binocular rivalry (Blake & Logothetis, 2002; Porta, 1593), has become a popular tool to explore visual consciousness scientifically and quantitatively. In binocular rivalry paradigms, dissimilar images presented separately to each eye may compete for perceptual predominance, while the physical stimuli themselves remain unaltered. Conventional binocular rivalry generates perceptual alternation via dichoptic stimuli that differ in at least one physical property (e.g. grating orientation) from their counterpart. Adjacent binocular rivalry methods, such as interocular grouping (Diaz-Caneja, 1928; Kovács, Papathomas, Yang, & Feher, 1996), Flicker-Swap rivalry (Logothetis, Leopold, & Sheinberg, 1996), travelling-wave rivalry (Wilson, Blake, & Lee, 2001), and continuous flash suppression (Koch & Crick, 2004) are part of an overarching family of multi-stable perceptual phenomena that cause an alternation of perception without a change of the physical environment. Rivalry paradigms have therefore been used to search for the locus of perceptual changes in the conscious state of the mind (Koch & Crick, 2004; Logothetis et al., 1996; Lumer, Friston, & Rees, 1998). Also, rivalry paradigms have been deployed as a clinical scientific tool in ophthalmology and optometry to determine and monitor eye dominance (Handa et al., 2004), as well as the behavior in clinical atypical groups, e.g. amblyopia (Harrad, 1996) or glaucoma (Tarita-Nistor, Samet, Trope, & González, 2020). Clinical cognitive scientists use rivalry paradigms as a tool to objectively measure differences between neuro-typical and neuro-atypical populations such as autism spectrum condition (Robertson, Kravitz, Freyberg, Baron-Cohen, & Baker, 2013), depression (Jia et al., 2015), attention-deficit (Amador-Campos, Aznar-Casanova, Ortiz-Guerra, Moreno-Sánchez, & Medina-Peña, 2015), as well as to investigate neural correlates of consciousness in brain-damaged patients (Spiegel, Laguë-Beauvais, Sharma, & Farivar, 2015).

For more than 100 years, multi-stable perceptual phenomena have been measured by first describing to the participant what they are expected to perceive and then asking the participant to report which of those descriptions they experience (with the exception of Alais et al., 2014). For example, the participant may be asked to report their experience in every moment of a trial by choosing among 2 to 5 alternative responses which is the closest match to the descriptions of perceptual states provided by the experimenter. For binocular rivalry, these perceptual states may be exclusive impressions of the images presented, piecemeal percepts in which portions of both stimuli are distinctly seen (Blake, O’Shea, & Mueller, 1992), superimposition of both stimuli simultaneously either equally in visibility (Liu, Tyler, & Schor, 1992) or with a predominance of one over the other stimulus (Yang, Rose, & Blake, 1992). Next to paradigms that measure which state was seen, the location of initial perceptual change, i.e. breakthrough point, can also be measured (Paffen, Naber, & Verstraten, 2008). Additionally, rapid, uni-directional, perceptual changes from one exclusive state to another across an area of the visual field are known as traveling waves of binocular rivalry (Wilson et al., 2001), and are distinct from so-called reversions, i.e. the change from one exclusive state to a mixed state and back to the previous exclusive state (Mueller & Blake, 1989).

During the pioneer period of rivalry research (1838–1920), the first scientific publications were subjective reports of single, experienced scholars (Hering, 1920; von Helmholtz, 1867; Wheatstone, 1838) who described in their own words how they experienced the competition. Breese (1899) used a kymograph to track the subject’s experience by instructing them to press separate keys corresponding to their current experience and noticed that exclusive percepts were seen, effectively creating the first 2-Choice rivalry response task (Breese, 1899). The implicit view of what we call “the essentially bistable nature of binocular rivalry” between the interocularly rivalrous stimuli should remain part of the measurement and intellectual traditions of rivalry researchers for many decades to come and is still prevalent in the field. Studies in recent years have begun to explore mixed perceptual states as their own perceptual experiences with distinct neuronal processing sites and mechanisms (Klink, Brascamp, Blake, & van Wezel, 2010; Skerswetat, Formankiewicz, & Waugh, 2018; Yang, Rose, & Blake, 1992) rather than classifying these states as transitory events between the bistable categories. Roelofs & Zeeman (1919) used also a kymograph but asked the participants to indicate with the stick which line orientation was predominant (Roelofs & Zeeman, 1919). This participant-via-experimenter-reporting approach was used in later years to collect rivalry data when working with young children (Lunghi, Morrone, Secci, & Caputo, 2016) or other populations that may not be able to indicate their perceptual changes reliably (Fox, 1965). The 2-Choice task was then used with different recording techniques, including key or switch presses that were dedicated for each of the two exclusive percepts (Whittle, 1965). Although Wheatstone (1838) noted that portions of the rivalrous stimuli may be seen(Wheatstone, 1838), the first explicit measurement that included “mixed percepts” were made by Cogan & Goldstein (1967) using a 3-Choice task (Cogan & Goldstein, 1967). Years later, a joystick tilt task (Hudak et al., 2011) was used to track rivalry alternations, in which ±75% or more tilt toward right and left were assumed to indicate either of the exclusive percepts. Other joystick approaches have used 1-dimensional reports (Fahle, Stemmler, & Spang, 2011; Naber, Frässler, & Einhäuser, 2011) with left and right categories and locations in between representing transitions between the categories. These in-between reports were not used to validate the nature of mixed perception states i.e. coarse or fine scale piecemeal or superimposition with either left, right, or equal predominance superimposition nor their spatio-temporal local variations.

To further disentangle the role of the two canonical mixed states, namely piecemeal and superimposition, and the exclusive states, Skerswetat, Formankiewicz, & Waugh (2018) developed a 4 choice method (Skerswetat et al., 2018). Similar to this work, Sheynin, Proulx, & Hess (2019) separated mixed states into equal states of piecemeal and superimposition, predominantly left or right, as well as exclusive states, creating thus a 5-Choice task (Sheynin et al., 2019). These methods allow an experimenter to track which state was seen and for how long it was seen. As a consequence, four measures of rivalry have been established since the first systematic investigation by Breese (Breese, 1899, 1909): absolute proportions of perceptual states, their mean and median durations, the number of perceptual alternations (Breese, 1899, 1909), and the distribution of exclusive state events (Levelt, 1965), which are typically well fit with gamma (Levelt, 1965) or lognormal (Zhou, Gao, White, Merk, & Yao, 2004) functions. Although individual rivalry experiences may differ substantially due to neuro-diversity within each population (Dunn & Jones, 2020), the stimulus properties (Brascamp, Klink, & Levelt, 2015), and the number of states that are considered within the N-Choice tasks (as outlined above), in general 10–30 responses per minute were typically recorded using standardized Choice tasks (Bosten et al., 2015). As for the report of the spatial location of initial perceptual change, probability maps are typically used (Paffen et al., 2008).

Alternative approaches to track rivalry changes have also been explored, including correlating perceptual changes during binocular rivalry as indicated via psychophysical choice tasks with eye movements (van Dam & van Ee, 2006) including optokinetic nystagmus (Hayashi & Tanifuji, 2012; Leopold, Fitzgibbons, & Logothetis, 1995), eye movements in relation to moving dots on a rivalrous stimulus as a marker of alternation between undefined perceptual states during rivalry (Hesse & Tsao, 2020), pupil size changes (Schütz, Busch, Gorka, & Einhäuser, 2018), blood-oxygenation variations in the brain (Lee, Blake, & Heeger, 2005), or visual evoked potentials (Brown & Norcia, 1997). For completeness of the discussion about rivalry paradigms it is worth noting that they have also been applied in non-human primates in which monkeys were trained to pull one of two levers that corresponded to an exclusive image under dichoptic conditions (Leopold & Logothetis, 1996).

The methods listed above have several shortcomings and lack several important features:

• A validated, personalized estimation of perceptual states and their boundaries

• An assumption that the experiences described by the experimenter represent the experiences for the participant, which may be especially problematic when neurotypical experimenters attempt to predict the experiences of neuro-atypical individuals

• Not all types of mixed perceptual experiences reported in the literature may be captured within each trial using a limited number of perceptual choices

• Perceptual experiences within mixed categories (piecemeal and superimposition) are assumed to be constant and any changes within those categories are not recorded

• Button-pressing report methods generate relatively few data (~30/minute) (Bosten et al., 2015) and are limited by dexterity, consequently the temporal resolution of those methods compared to the actual experience may be low

• Some previous studies have used replay or mimic approaches of physically changing stimuli to validate response reliability (Goldstein & Cofoid, 1965; Lumer et al., 1998), but these reliability checks have not been established as standard practice yet

In recent years, so-called no-report paradigms have become a popular tool to study visual consciousness (Einhäuser et al., 2008; Naber et al., 2011). Using physiological, involuntary measures rather than additional action to measure changes of interocular visual suppression allows the investigator to search for the locus of visual consciousness in humans (Frässle, Sommer, Jansen, Naber, & Einhäuser, 2014) and monkeys (Kapoor et al., 2022). No report paradigms attempt to address issues of report-paradigms, which were summarized by Naber et al. and who identified four problems during the study of visual consciousness using binocular rivalry (Naber et al., 2011, page 3): “…motor-act of reporting itself may affect perception, […] report mode might restrict response possibilities, […] very brief dominance periods of one percept might not suffice to trigger a report; and […] catch trials might not mimic the entire phenomenology of rivalry.” We argue that no-report paradigms only address the first of the four shortcomings sufficiently, whereas the other three points still have limitations with no-report paradigms. We therefore explicitly focus on our critique of the question of whether they are an improvement to other report-paradigms not about other concerns regarding their ability to search for neural correlates of consciousness, which are expressed elsewhere (Overgaard & Fazekas, 2016). To begin with, no-report paradigms can be subdivided into pupil size change, optokinetic nystagmus approaches, which are based on passive reflexes and a gaze direction approach, based on an active fixation of moving dots.

As passive no-report paradigms use report-paradigms to establish a relationship between psychophysical and physiological outcomes first, they use conventional report-paradigms and thus suffer their limitations. For example, in a study comparing pupil-size changes during 2Choice-button press and 1-D joystick paradigm, Fahle, Stemmler, & Spang (2011) used orthogonally orientated gratings, one of which had higher luminance than the other to elicit a pupillary response when being seen. The authors showed that pupil size changes pre and post the reported indication corresponded well with the reported results using button pressing or joystick tilting when factoring in reaction time. Unfortunately, the authors did not repeat the experiment without the report paradigm by only measuring pupil size changes to confirm that similar statistics of pupil size changes across testing time occurred. The no-report association of the pupils here is therefore based on the specific report paradigms. The participants were instructed to press two buttons in one task and move a joystick between two poles hence they did not take all known perceptual states into account (Klink, Brascamp, Blake, & van Wezel, 2010; Yang, Rose, & Blake, 1992) nor their dynamic changes within mixed states (since the individual introspection was unknown, the assessment of the joystick movement is problematic as even 50%−50% piecemeal percept might be due to two coarse of piecemeal or due to many smaller portions of piecemeal, which may cause judgement errors by the observers). They also assume that the decision criteria for pressing one of two buttons and across time is the same. A replay mimic was used but it was based on the report of the true rivalry trials therefore also did not address the different types of mixed perception. Using the joystick task to only >60% joystick changes in one of the two were included in the analysis whereas χ.60% were excluded, which is an arbitrary criterion and may not include inter-individual difference in perceptual state judgement. The use of continues joystick approach was however an important improvement as the authors recognized the gradual nature of rivalry alternations across time and it allowed in principle the measurement of smaller, subtle changes between the bistable categories.

In the same year, Naber et al. (2011) also utilized measurements of pupil size changes via difference in luminance between stimuli while using a button pressing and 1D joystick movement methods. Furthermore, those authors also introduced optokinetic nystagmus (OKN), i.e. eye movements evoked through moving stimuli to ensure stable binocular fixation (slow phases movement) or to return to the primary gaze position (fast phases movement) as another physiological predictor of perceptual changes. However, the general protocol was the same as for the pupil approach: first measuring OKN and pupil size changes with report-approaches and then using a physical replay of their perception mimicking exclusive and piecemeal states. Note that the latter were unidirectional changes from one to another stimulus using a sinusoidal movement pattern, which may reflect traveling wave-like changes (Wilson et al., 2001), but other traveling wave patterns or speeds were not reflected and again, other piecemeal and superimposition perceptual states were not taken into account for either the rivalry or replay phases of their pupil and OKN experiments.

Both pupil size changes and OKN approaches utilize passive reflexes. In contrast an active approach was developed by (Hesse & Tsao, 2020) in an electrophysiological study in monkeys. The monkeys were trained to maintain fixation on a jumping spot using two rivalrous stimuli. While following the fixation spot, viewing conditions were either physical, monocular alternation of the stimuli, or dichoptic perceptual alternations with no physical change of the stimuli. During the binocular rivalry condition, a fixation spot was shown to each eye at different positions on the screen i.e., when perceiving the right eye’s stimulus, the monkey may perceive exclusivity of the right eye’s fixation spot and saccade to it, ignoring the fixation spot in the left eye. The method is an elegant way to infer perception of one eye’s stimulus from eye movement patterns without active report. However, as for the other no-report paradigms mixed states and their changes are ignored and piecemeal perception may not be inferred from the report data. Furthermore, fixation task and the search after disappearance is another active task factor which may confound reporting as traditional report-paradigms i.e., button pressing, joystick or in monkey lever pulling.

In conclusion, the use of joystick report-paradigm while tracking physiological responses first and then using those reflexive physiological measures alone as predictor of rivalry alternation in a no-report phase is an elegant way to avoid biasing rivalry experiences with active action confounder. No-report paradigms can thus be used to infer that a perceptual change has occurred and has since its development sparked exciting research (Kapoor et al., 2022; Naber et al., 2011). However, these paradigms have still limitation for the search of visual consciousness (Overgaard & Fazekas, 2016), and as we argued in the section above, current no-report paradigms do not provide a validated estimate of what has been seen i.e. visual introspection.

The aim of the current paper is to introduce a method that is designed to address these shortcomings. InFoRM (“Indicate-Follow-Replay-Me”) here applied to binocular rivalry, “InFoRM: Rivalry”, is a novel 4-phase-method that dynamically tracks and validates intrinsic visual experiences, allowing validated estimations of introspection, and novel analysis of perceptual alternation dynamics due to its high temporal resolution (Figure 1). Prior to the start of the experiment, the participant is asked to wear glasses throughout the experiment, which are in fact polarized glasses that can be used for binocular and dichoptic stimulus presentation. This step was introduced from the beginning to avoid any hint as to when rivalry would be measured. Importantly, InFoRM also makes no assumptions about the multi-stable nature of binocular rivalry.

Figure 1:

Scheme of InFoRM: Rivalry protocol. During Indicate-Me, participants explored the stimulus-space for 60sec, moving the joystick to modify binocular-non-rivaling stimuli in real-time and simulate six canonical rivalry states. During Follow-Me, participants matched perceptual reports for physically changing binocular-non-rivaling-stimuli in four author-created rivalry-trials and four self-generated trials illustrating canonical rivalry states from Indicate-Me. During Rival-Me, participants reported their perception during eight 60sec-trials of dichoptic-rivalry. During Replay-Me, participants’ responses during the eight Rival-Me dichoptic-trials were used to generate physically changing binocular stimuli, which validated their individual perceptual-state-space.

Phase 1: “Indicate Me”- Visuomotor practice

InFoRM: Rivalry requires participants to indicate their perception of a physical binocular stimulus via joystick (or other data input device) while viewing a screen. Participants explore the change of the presented physical binocular stimulus caused by their own joystick movements, thereby exploring and learning the joystick-stimulus relationship. Phase one trains participants to understand the relationship between change of input device, here a joystick, and changes in the appearance of the stimulus. In this phase, the experimenter highlighted six perceptual states associated with rivalry, among the many thousands of states generated via physical changes of the stimulus.

Phase 2: “Follow Me”- Visuomotor testing

In Phase 2: Follow-Me, participants view ongoing changes of the physical binocular stimulus and move the joystick to follow the dynamic changes, thereby confirming the learned joystick-stimulus relationship. The stimuli presented in this phase mimic actual rivalry trials and serve as training for the participant in the report paradigm and most importantly also validate their own indication generated during Phase 1 for principal states that may occur during binocular rivalry. Phase two accomplishes two objectives: participants indicate their perception of dynamically changing stimulus while also confirming their specific perception of the six perceptual states from classic rivalry studies.

Phase 3: “Rivalry”- Actual binocular rivalry testing

In Phase 3: Rival-Me, participants view a dichoptic rivalry stimulus that is physically unchanging and move the input device to report their experience of perceptual-rivalry induced. As the task has not changed, the participant remains blind to the change from binocular to dichoptic stimulus presentation. The overarching goal of phase three is to collect continuous binocular rivalry data.

Phase 4: “Replay Me”- Physical replay of binocular rivalry testing

In Phase 4: Replay-Me, participants view a physically changing binocular stimulus and move the joystick to follow the stimulus, but this time the physical changes are generated by the responses generated by the participant themself during Phase 3. Again, participants were blind to the change from dichoptic to binocular stimulus presentation. This phase validates their individual perceptual-state-spaces by replaying the physically changing stimulus input based on each observer’s true rivalry input during phase 3 to estimate the agreement between these data, i.e. the more similar the data of phase 3 and phase 4 for an observer, the greater the reliability of the inputs during true rivalry.

We show that the InFoRM: Rivalry method generates validated estimates of perceptual state experiences that can be used to analyze rivalry dynamics. The Phase 2 data provide new insights in introspection and judgement of perceptual states boundaries. Perceptual rivalry data gained during Phase 3 can be analyzed with conventional measures, i.e. relative and mean proportions of perceptual states, perceptual alternation rates (Breese., 1899), and with Gamma functions fit over perceptual exclusive states (Levelt, 1965; Zhou et al., 2004). Moreover, the high-temporal resolution of InFoRM: Rivalry allows investigation of the dynamics of perceptual alternations using analysis techniques borrowed from eye tracking research. We also gain deeper insights into the changes within mixed states that are not possible when each state has a single report choice. Lastly, the replay data in Phase 4 test how accurately the participants were able to follow their own rivalry dynamics in a physical task and validates the rivalry introspection data collected in Phase 3.

2. Methods

The experiments were carried out in the facilities of Northeastern University, Boston. Written and verbal information about the project were provided in advance to the participants and they gave written informed consent before taking part. Ethics approval to conduct the experiments on human participants was in line with the ethical principles of the Helsinki declaration of 1975 and ethics board of the Northeastern University. Participants were recruited from Translationla vision laboratory as well as from the undergraduate population at Northeastern University, Boston. Undergraduates received course credit towards the completion of their Introductory Psychology course in exchange for their participation.

2.1. Equipment

Stimuli were presented on a LG 3D monitor 1980*1080 pixels (55.7 pixel/°) with a framerate of 60 Hz at a viewing distance of 150cm using a Dell computer (Optiplex 7060). The participants wore radially-polarized LG cinema 3D glasses (AG-F310) and provided responses with a Logitech Extreme^™ 3D pro (Logitech Europe S.A.) joystick.

2.2. Stimuli

Matlab (2019b) software was used to generate all the code for the experiments in combination with Psychtoolbox version 3.0 (Brainard, 1997; Pelli, 1997). Prior to the experiment, the monitor was gamma-corrected using a Photo Research SpectraScan 655 (Norway) spectrophotometer. Viewer crosstalk (i.e. leakage of the luminance of one eye’s image, to the image of the opposite eye’s image) was minimized with Psychtoolbox’s StereoCrosstalkReduction and SubtractOther routines to minimize the subjective visibility of a 100% contrast 2 c/deg sine grating presented to one eye that was patched and a mean luminance field presented to the other eye that was used to judge crosstalk.

Non dichoptic grating stimuli:

The circular aperture of sinewave gratings had 2° diameters, and 2cycles /° spatial frequency, which provides high perceptual alternation between the stimuli thus favors exclusive over mixed perception using a 3 choice rivalry task (O’Shea, Sims, & Govan, 1997) without biasing the study towards mixed perception as it is known that larger stimuli generate predominantly mixed percepts (Blake et al., 1992). To change the degree of perceptual states, we varied contrasts uni-and bilaterally as it has been shown that superimposed mixed increases with lower bilateral contrasts (Brascamp, van Ee, Noest, Jacobs, & van den berg, 2006; Liu et al., 1992) while exclusivity is enhanced when unilaterally changing contrasts (Brascamp et al., 2015; Levelt, 1965). The gratings were obliquely (135° and 45°) orientated. The Michaelson-contrast conditions for the stimuli were bilaterally 10% or 50%, or unilaterally 10% vs 50%. A white central spot of 0.1° diameter was used as a fixation marker. A circular fusion lock, i.e., a surrounding ring (width of 2 pixels) perceived by both eyes at the same location that will be fused and thus preventing eye misalignments, surrounded the stimuli with 3° radial distance from the center of the stimuli. Stimuli were presented on a grey background with a mean luminance of 61.9 cd/m² in a windowless room with constant artificial lighting. An alpha blending (i.e., computational combination of color values of pixels from two images) procedure was used to merge two orthogonal gratings presented within a gaussian window and were updated in real time to joystick movements.

Changes to the physical stimuli were created with band-pass filtered noise that was used to spatially combine the orthogonal gratings. Random gaussian noise was filtered with a log cosine filter whose peak spatial frequency (F_peak) was varied with joystick movements along the vertical axis. F_peak varied in log steps from 1 cycle per image at the minimum vertical joystick position (closest to the participant) to the Nyquist limit at the maximum position (farthest from the participant). This created regional blobs whose size varied from half the stimulus size when the joystick was at the near position to 1 pixel when the joystick was at the far position. A blob size of 1 pixel should create a transparent stimulus, however local pixel clumping meant that small regions of each patch were formed. To avoid this, when the blob size was <2 pixels, transparency was created with alpha blending the left and right eye images.

The noise was scaled to the range −1 to +1 with zero mean. A cut off value between −1 and +1 was used to assign pixels to either the 135° or 45° grating. The cut off value was varied in linear steps with joystick movements along the horizontal axis. Areas of the bandpass filtered noise with values below the cutoff (darker noise areas) were assigned to the 135° grating and values above the cutoff (lighter noise areas) were assigned to the 135° grating. Thus, when the joystick was fully to the left, the blended image was a uniform 135° grating; when it was fully to the right, the blended image was a uniform 45° grating; when it was fully near the blended image was 2 large blobs one with a 135° grating, the other with a 45° grating; when it was fully far the blended image was a transparent 135° and 45° grating. For transparent stimuli (blob size<2 pixels) the contrast of the left and right eye images was linearly scaled from 0 to 1 and 1 to 0 respectively, corresponding to the joystick horizonal position.

A new noise sample was created at the start of each test period.

Dichoptic grating stimuli:

Polarized glasses (LG Cinema 3D, AG-F310) ensured dichoptic representation of the stimuli via a 3D monitor in which pixels on even and odd rows are presented to opposite eyes. All other properties were the same as for the non-dichoptic grating stimuli.

2.3. Participants

Stringent Covid-19-related security measures were taken to conduct this research: All participants except author J.S. were students at the Northeastern University, MA, USA. As part of the university policy, students and staff were required to wear facial and nostril masks while being on campus and while being indoors, including labs. Also, a mobile app was required to be installed and activated while being on campus. In addition to those requirements, N95 facial masks and transparent face shields were worn by the experimenter. Disposable surgical gloves were worn by the experimenter and the participant. Affected materials and surface were sterilized using alcohol-based solutions, small items such as polarizing glasses were additionally disinfected using a 20min UV sterilizer.

No participants, except author J.S., had experience in rivalry experiments nor were they aware of the design of the study. Initially 30 participants took part in this study (Table 1). General exclusion criteria from the main analysis were diagnosis of autism, attention deficit disorder, epilepsy, migraine, dyslexia, or any other mental health condition.

Table 1.

Demographic and optometric screening data. Mean age, sex assigned at birth, handedness, Miles eye dominance, mean visual acuities and ranges [Snellen acuity] for right (OD), left (OS), and both (OU) eyes, and mean stereoacuity in arc seconds and range are depicted.

Total # participant n/included n	Age [years]	Sex [f/m]	Handedness [Right/Left]*	Eye dominance [Right/Left]*	Visual acuities@4m [20/_]	Stereoacuity@41cm [arcsec]
30/28	18.6 [17–34]	15/13	25/2	18/9	OD: 14.5 [10–20] OS: 15.6 [13–20] OU: 14.2 [10–20]	40 [40–50]

^*Missing information for one participant.

Prior to the experiments, an optometric screening was carried out by an optometrist (author J.S.) or trained research assistant to ensure normal binocular vision. Specifically, all participants had normal or corrected-to-normal monocular visual acuities measured in 4m distance to a retro-luminant ETDRS chart of at least 20/20, a binocular acuity that was the same or higher, and reported that they had no ocular-related surgery or treatment in the past. Normal binocular vision was then indicated by measuring stereoacuity using the Titmus test (stereoacuity ≤ 100arcsec). A Worth 4-Dot test for the distance of 1.5 m was carried out to test for central interocular suppression and all participants perceived 4 lights, indicative for no central suppression. The Miles eye dominance test was carried out to determine the eye dominance. Here, the participants were asked to fixate the experimenter’s right eye through a small gap made by folding their hands. We also asked whether the participant was left or right-handed. Two participants were excluded (one attention-deficit, one epilepsy) from the main analysis. Ten participants wore glasses, three contact lenses, fifteen did not wear spectacle correction.

2.4. Psychophysical Procedure

After the optometric screening, the participants were brought to the lab in which the rivalry experiment was carried out. First the chair, chin and forehead rest were aligned so that the participant was sitting comfortably. The joystick was placed on the right-hand side and was used by all participants with their right hand. Polarized glasses were worn throughout the experiment without further explanations to ensure that participants remained blinded to the task and to keep the contrasts for all phases constant.

As shown in Figure 1, during Phase 1 (“Indicate me”), participants first freely explored the relationship between joystick position and physical stimulus change in all joystick directions during a training trial, followed by the actual session in which the participant was asked to indicate and explore the space in which each state of the six predefined rivalry states that had been reported previously, namely exclusive left and right (Breese, 1899), piecemeal (Wheatstone, 1838), equal superimposition (Liu et al., 1992), and perception of superimposition and piecemeal with either left or right predominance (Sheynin et al., 2019) was being perceived (i.e. one state at a time, 10 sec/state). For example, the participants were instructed verbally via experimenter and via written text displayed on the screen to indicate and explore the joystick space for a left exclusive percept by a left-of-center movement of the joystick. The actual indication duration for each state was 10sec, then the participant had to press the joystick trigger button to continue with the right exclusive state etc., hence the total phase one took 1min. In some instances, an additional session was carried out if the participant was still unsure about the joystick perceptual state relationship. During Phase 2 (“Follow me”), the participant was instructed to use the joystick and follow the perceived changes of the stimulus as soon as they were noticed by tilting the joystick into the dedicated directions learned during Phase 1 while observing dynamically changing physical stimuli. Note that the participant was being made aware that the appearance of the stimulus may change. Each trial was initiated via joystick pull and release of the trigger, lasted 60secs and were stopped abruptly by the program. To ensure the task was understood, the experimenter observed whether the approximated joystick location and stimulus appearance were aligned, e.g. exclusive left stimulus means joystick left tilted, equal superimposition means joystick tilted forward etc. and verbally explained what movement was expected during the first alternations of the first trial. All participants were able to perform the task. After the last trial, a break was given and followed by Phase 3 (“Rivalry”) during which the task remained the same as during Phase 2 except that now perceptual rather than physical changes were being tracked. The participant was not informed that control of the stimulus control had changed. After completion of Phase 3 and another break, Phase 4 (“Replay me”) started with the same task as for Phase 2.

The initial Phase 1 during the first run consisted of two 1min trials to first familiarize the participant with the joystick, and then for the subsequent contrast condition only 1 min trial, Phases 2–4 included eight trials/phase. Three contrast conditions were used, namely 0.1vs0.1, 0.5vs0.5, and 0.1vs0.5 counterbalanced between the eyes. Stimulus orientations were also counterbalanced between trials. Each contrast was used for all 4 phases of InFoRM, the order of contrasts was randomized between participants. The completion of the entire experiment, including the screening, took approximately 120 min.

2.5. Data analysis

Raw data consisted of 3600 data points (60Hz data sampling * 60seconds testing) in total for horizontal and vertical joystick vectors for each phase and was stored in customized .mat files.

2.5.1. Data processing during the experiment

Phase 1) “Indicate me”

The horizontal and vertical joystick data for each of the six perceptual states lasted 10sec/600 data points and were stored in a .mat file and used as ground truth for phase 2).

Phase 2) “Follow me”

Author J.S. generated actual binocular rivalry joystick data for 60sec/3600 data points for varying contrast conditions and stored them as training data. Phase 2 consisted of 4 trials of these training data and 4 trials of the individual’s Phase 1 data, each state’s input randomly connected within a trial. All training and Phase 1 trials were then randomized. The participant had thus the opportunity to train to follow an actual rivalry experience as well as an opportunity to indicate each of the six states, generate by the participant themselves. The data were stored in a .mat file.

Phase 3) “Rivalry”

Eight trials, each consisting of 60sec/3600 data points were collected per contrast condition and stored in a .mat file.

Phase 4) “Replay me”

Data from Phase 3 were read in and used to generate physical stimulus changes during Phase 4. The eight trials of joystick data were then again stored as separate .mat file.

2.5.2. Post-experiment data analysis

A customized Matlab (Version 2021a) program was written to analyze the raw data.

Phase 3) “Rivalry” data analysis

Conventional binocular rivalry outcome measures

The joystick data generated during Phase 3 and the classification data as described above were used to calculate traditional measures of binocular rivalry. Here, we begin by describing conventional rivalry outcome measures and how we derived those. The results are used to compare InFoRM rivalry data with that of conventional report-methods.

Relative proportions, mean and median durations, perceptual alternation rates

A single raw data point represents a (x,y) joystick location in a period 16.7ms (i.e. 60Hz sampling rate) which indirectly captures visual experiences for the same temporal. We used the classifications from Phase 2 to assign each Phase 3 “Rivalry” data point to a perceived state, which allowed us to calculate the mean duration and the relative proportions of each state per trial. We averaged those results across trials for each participant. Also, changes of classified states were counted for each theoretically possible alternation type, e.g., exclusive left to piecemeal, piecemeal to exclusive left etc. for each trial and then averaged across trials for each condition. Next to the breakdown of all single alternation types, we generated three alternation categories: 1) all flips, i.e., total sum of all occurring alternations; 2) exclusive to mixed states alternations and vice versa, i.e. sum of flips between exclusive and mixed states; and 3) mixed to mixed alternation, i.e. the sum of all within mixed perceptual alternations. The contrasts during the low vs high condition were counterbalanced and we arranged the data for post-processing accordingly. Additionally, boxplots were used to indicate median, interquartile ranges, extreme and outlier values.

Analysis of perceptual phase distributions

For each trial, contrast condition, and participant, data were first normalized by dividing the phase durations by the relevant mean. These normalized data were then combined across participants and contrast conditions. The perceptual phases are presented in the following form using a gamma distribution:

$$f(x\mid \alpha ,\beta )=\frac{1}{{\beta }^{\alpha }\Gamma (\alpha )}{x}^{\alpha -1}{e}^{\frac{-x}{\beta }};x>0,\alpha >0,\beta \ge 0$$

where is the gamma function with Γ(α) “shape” α that represents the skewness of the distribution, β scales the distribution along the abscissa and the number of perceptual events x (Levelt, 1965). The coefficient of determination (R²) has been used in previous studies(Logothetis et al., 1996) as an indicator of how well observed data fit a predicted model.

Also, we analyzed the area under the curve (AUC) of the Gamma function, calculated the peak of each function, latency (X-peak), and its amplitude (Y-peak) (Skerswetat, Bex, & Baron-Cohen, 2022). To be comparable with previously reported data, the range of the x axis went from 180ms to 4000ms.

Novel InFoRM outcomes for binocular rivalry

Next, we are describing the novel rivalry outcome measures generated by the InFoRM paradigm.

Classification of perceptual states generated during phase 2) “Follow me”

Joystick indications during the phase 2 in which each of six principal states were shown, and contained the data generated by each individual during the “Indicate me” phase 1. Each of the 6 states consisted of 600 data points. These data were collected in randomized order between states within a trial and further randomized with data that mimics actual rivalry behavior (seed dichoptic data generated by author JS). This way we ensured that a participant had the opportunity to learn to follow realistic dynamic changes on the one hand but also had to indicate the location of each of the six canonical states that were generated by themselves during phase 1. After the experiment, we extracted the joystick data that corresponded to the presentation of each of the 10sec long states (Figure 2 A). Then, we used a customized Gaussian Mixture model to classify the most likely area for each perceptual state within the two-dimensional joystick space. Specifically, clusters of 600data/trial were combined across trials (4 training and 4 phase 1 embedded during Follow Me phase 2) were used to create a 2D gaussian probability density function for each of the six states (Figure 2 B). The maximum likely probability at each joystick location within the 2D joystick was then calculated to find each state’s distinct boundaries for each observer and contrast condition (Figure 2 C).

Figure 2:

Classification maps of visual introspection. Example of one observer’s joystick indication extracted during ‘Follow Me’ (A), its topography of value density for each state (B), and its classification results of the most likely perceptual states (C). The spaces in C) refer to six canonical states: Very dark blue (west), dark blue (east), light blue (middle to south), green (north), orange (northwest), and yellow (northeast) refers to left exclusive, right exclusive, piecemeal, equal superimposition, left-predominance superimposition, and right-predominance, respectively. Results for all individuals and the average across observers for the high condition are shown in D). Boxplots and one-way ANOVA results of the relative size of the states averaged across trials and contrasts are depicted in E).

Eye dominance scores

A general eye-dominance score, i.e., percentage of time spend in left-vs. right-of-joystick-center was generated for each trial, and averaged those across trial, participants, and conditions. This score may be helpful for clinical assessment of overall eye dominance in conjunction with the relative values for each state.

Perceptual Velocity: Rivalry “Fixations”, “Tremors”, “Micro-saccades”, “Saccades”

Joystick movements are captured by 60Hz (or higher) data of horizontal and vertical locations that may change across trial duration, depending upon the perception was stable (i.e. no joystick movement) or perception alternated (i.e. joystick tilt changed). Here, we apply eye tracking techniques used to classify different eye movement subtypes. First, we calculated the gradient for horizontal and vertical joystick changes, and then the mean velocity (mean trial speed) and its standard deviation (1 SD trial speed) for each trial to classify the following four categories of joystick location changes within each trial (see example Figure 5A):

Figure 5:

Perceptual velocity during binocular rivalry. (A) Scheme of classification space and a hypothetical joystick indication with labeled velocity markers. (B) Example trial state changes (black line) across trial. Trial mean (green line), SD (red line) and the actual velocity changes (blue line) are depicted. C) depicts the relative proportions and D) the absolute number of events of the four change categories, namely Stable perception (blue), rivalry tremor (cyan), micro-saccades (green) and saccades (yellow) averaged across trials and contrast conditions. One-way ANOVAs and significant differences found via planned comparisons are depicted as well (*p<0.5, **p<0.01, ***p<0.001). E) Colormap visualizes the (lack of) change of velocity for all trials, conditions, and participants. D) Fast Fourier Transform of velocity for each trial, contrast conditions, and participants.

‘Stable perception’ (Speed =0), ‘Rivalry tremor’ (Speed > 0 & < mean trial speed), ‘Rivalry Micro-saccades’ (Speed >= mean trial speed & <= 1 SD trial speed), and “Rivalry Saccades’ (Speed > 1 SD trial speed). Then, we averaged those speed categories across trials and contrast conditions, used a one-way Analysis-Of-Variance (ANOVA), using Matlab’s anova1 function, to calculate the effect of those speed categories. For planned comparisons between each category, we applied the multcompare function. A Fast Fourier Transform (FFT) was then used to calculate the distribution of perceptual dynamics with Gamma function fits. AUCs of the Gamma function, latency (X-peak), and amplitudes (Y-peak) were calculated.

Within mixed states analysis

To explore the changes within each of the four mixed states, we first classified the perceptual states reported in each trial to confirm that piecemeal, equal superimposition, left or right predominance superimposition had occurred. If so, the speed of perceptual change method described above was deployed for each mixed state subtype. We then summed the number of each of the events and calculated their mean duration for each trial, participant, and contrast condition. Lastly, we found that the trends were the same for each individual mixed perceptual state, so we averaged the results across mixed perceptual states and used one-way ANOVAs and planned comparisons to investigate the effects of rivalry subtypes on the number of total events and their mean durations.

Blob size analysis during mixed perception

We calculated the blob size from the joystick y axis position of each stimulus input during Phase 3) within the mixed perceptual reports and analyzed their distribution and mean size. Specifically, we extracted all vertical raw joystick data that were classified as piecemeal for each trial, participant, and contrast condition.

Replay analysis using InFoRM method

We compared joystick reports from ‘Rivalry’ and ‘Replay’ phases to validate individual rivalry dynamics using pairwise data comparison. First, we classified perceptual states using the classification maps generated during phase 2 for each phase 3-true rivalry and phase 4-replay data point for all trials and observers and conditions. Then, we used Matlab’s pdist2 function and applied a Hamming distance algorithm comparing each replay data point with that of the true rivalry datapoint in a moving window loop. Finally, we fitted individual’s results for each contrast condition with an exponential function to estimate agreement between true rivalry and replay data for each observer and their perceptual delays as measured via reaction time. As a result, we gained similarity in percentage between true and replay rivalry data via the y-axis intersection with the asymptote and the reaction time via x-axis intersection (see supplementary materials).

3. Results

Due to the large amount of data and techniques applied, we focus this work on averages across contrast conditions, and will report between contrast conditions only in the eye dominance score section or in designated parts of the Supplementary Materials.

3.1. Phase 2: Follow-Me Classification

InFoRM: Rivalry’s classification of each state is to the best of our knowledge the first method to generate a validated estimate of a person’s internal representation of perceptual states, here mimicking alternations during a binocular rivalry task. Figure 2 A shows the joystick positions adopted by a representative participant during Phase 2, in which the participant was asked to move the joystick dynamically as accurately as possible to capture which state was seen, namely left exclusivity (very dark blue (west)), right exclusivity (dark blue (east)), piecemeal light blue (middle to south)), equal superimposition green (north)), superimposition with left predominance orange (northwest)), and superimposition with right predominance yellow (northeast)). Clusters of the joystick indications data are shown for each of the 6 Gaussians in Figure 2 B and were classified using a Gaussian Mixed Model by calculating the maximum likely probability at each joystick location, resulting in the classification of six distinct states for each observer (Figure 2 C), examples shown for all 28 observers for the high contrast condition (Figure 2D). It is noteworthy that peak values as shown in the example in Figure 2B are due to varying degree spatial accumulation of data for each perceptual state, i.e., the more the data were spread the flatter the topography. Figure 2 E shows the area of each cluster corresponding to the joystick space for each perceptual state averaged across contrast conditions. A one-way ANOVA revealed a significant difference between perceptual state sizes (Figure 2 E). Interestingly, although the task was the same for each contrast condition, substantial differences in state sizes were found (see supplementary information).

3.2. InFoRM: Rivalry replicates standard rivalry outcomes

InFoRM: Rivalry is to our knowledge the first paradigm that continuously measures dynamic perceptual states within a trial that can be classified with a priori methods. There was a significant difference in the proportions of joystick intervals that each perceptual state was experienced (see Figure 3A), with significantly greater reports of left and right exclusivity compared to the other states. Previous studies using comparable stimulus configurations that reported intervals of exclusive versus mixed perceptual states showed that mixed or exclusivity states are perceived approximately 50% of the test duration (Hollins, 1980; Skerswetat, Formankiewicz, & Waugh, 2016), which is in line with the results gained with InFoRM: Rivalry (52% ±9 (Standard deviation (SD)) exclusivity and 48% ±9 mixed perception). Moreover, we show that piecemeal 12% ±12 occurred less frequent than the sum of all superimposed states 35 % ±19 (Figure 3 A). The results of the overall proportion of each report are accompanied with a significant effect for the mean duration of each reported experience (see Figure 3B). The mean durations of exclusivity reports were 1.2 sec ±0.3 longer than the mean durations of reported of mixed perceptual experiences (0.5 sec ±0.2). On average, there were more perceptual alternations between exclusive and mixed states than alternations between mixed perceptual reports (see Figure 3C). A full overview of all perceptual alternations can be seen in the supplementary materials. Previous studies have shown that the distributions of exclusive events during binocular rivalry are well fitted with a Gamma-function (Levelt, 1965; Zhou et al., 2004), which we replicate using InFoRM (Figure 3D) indicated with R² of 0.69.

Figure 3:

Summary of traditional measures for binocular rivalry derived from an analysis of InFoRM data. Averages across trials, participants, and conditions for A) relative proportions B), mean durations C), sum of all perceptual flips, and D) Gamma function fits to normalized exclusive percepts with their respective histograms. ‘L’, ‘R’, ‘PM’, ‘ES’, ‘LS’, and ‘RS’ in A-C refer to sum of left and right exclusivity, piecemeal, equal superimposition, superimposition with left predominance, and superimposition with right predominance, respectively and ‘All’, ‘EV to Mixed’, and ‘Mixed to Mixed’ in C) to all flips, flips from exclusive to any non-exclusive state, and flips within mixed states, respectively. The scattered dots indicate data for each individual, squares depict the means, boxes the interquartile ranges (25th-75th percentiles), horizontal lines within each box the medians, whiskers extend to the extreme values, outliers are plotted outside the whiskers. One-way ANOVAs were performed to test for difference between states or alternation categories. D) depicts the Gamma function (black dashed curve), the histogram, and the following parameters: shape α and scale β parameter of the Gamma function, number of events N, coefficient of determination R², area under the Gamma function curve (AUC) from 0.18 to 4 along x axis, X/Y peak of Gamma function.

We further investigated each possible single flip type in a separated heatmap table that provides a complete breakdown of all types (Supplementary Materials).

3.3. Eye dominance scores

Eye dominance can be estimated from the overall horizontal bias of the joystick. We here report individual data for each contrast condition to demonstrate that InFoRM detects predominance biases during binocular rivalry, which have been used as tool for determining eye dominance in various clinical populations (Bossi, Hamm, Dahlmann-Noor, & Dakin, 2018). We analyzed the total portions of joystick movement that were either to the left or right of the joystick center as an eye dominance measure. We also used the relative proportions of exclusivity and predominance during superimposition to further investigate sensory eye dominance between these perceptual states. Figure 4 shows eye dominance for all 28 participants and an average for each condition (black bar). Figure 4A and D show that there was an overall left bias (more positive and negative data points) for low and high contrast conditions, presumably due to the use of the right hand which may make a leftward movement more comfortable. The low vs high contrast condition (Figure 4 G) results show for most individuals a bias toward the high contrast stimulus, as predicted by Levelt’s first and second law of rivalry (Brascamp et al., 2015; Levelt, 1965). These measures may be useful for clinical screening and monitoring of populations with disrupted sensory dominance along monocular (exclusivity, Figure 4 B, E, H) and binocular (superimposition, Figure C, F, I) stages of the visual pathway, such as people with amblyopia (“lazy eye”) or asymmetric vision loss caused by eye diseases such as age-related macular degeneration or glaucoma.

Figure 4:

Eye dominance scores for Low (A-C), High (D-F), and Low vs High (G-I) contrast condition. Individual differences of relative proportions between left minus right eye’s total values (A, D, G), exclusive visibility (B,E,H), and left vs right superimposed predominance (C,F,I) are shown either within ±1SD (green), between ±1SD within and ±2SD (orange), and ≥ ±2SD (red). Black bars on the right of each graph depict the mean across participants within condition.

3.4. Perceptual Velocity: Stable perception and Rivalry- ‘Tremors’, ‘Micro-saccades’, and ‘Saccades’

As has been reported for gaze patterns of behavior (e.g. Yarbus, 1967), we noted that joystick rivalry reports contained periods of relative stability (analogous to fixations) separated by transitions to new report locations that were gradual (analogous to smooth pursuit eye movements) or rapid (analogous to saccadic eye movements) (Mahanama et al., 2022). To quantify the dynamic properties of the joystick reports, we adapted analytical methods developed for eye tracking data. Across trials, conditions, and participants, medians of relative proportions for stable experiences, rivalry-tremors, -micro saccades, and -saccades were 71.2, 9.7, 7.9, 12.1% ± 2.0, 1.2, 0.4, 1.0 standard errors, respectively. Gamma fits for each FFT across trials, conditions, and participants showed a median scale of 0.8Hz and R² of 0.29.

3.4.1. Analysis of changes within perceptually mixed states

The InFoRM method enables the analysis of perceptual changes within mixed states, that are not tracked with other methods. We deployed the speed analysis introduced in Section 3.4 for piecemeal and the three superimposed states. We found that stable perceptions were fewer in total number but longer in their main durations (Figure 6 A and B), indicating that the visual system tends towards perceptual stability. The Gamma fits for each FFT as calculated for phases of either piecemeal, equal, left-or right predominance superimposition had a median scale of 0.997, 0.994, 0.960, and 0.942 Hz, respectively, averaged across trials contrasts, and participants (see also Supplementary Material).

Figure 6:

Boxplots of averages across contrast conditions and mixed perceptual states, i.e. piecemeal, equal, left- and right-predominance superimposition. A) depicts the sum of events (y axis) for each perceptual subtype (x axis) as well as their respective means (B). One-way ANOVA results, planned comparisons (*p<0.05; **p<0.01; ***p<0.001), and number of participants N are shown. Black squares depict the mean values. C) Illustration of the relationship between joystick tilt and estimated blob size (thus number of blobs/degree) when moving the joystick forward or backwards. D) Example histogram of estimated blob sizes during piecemeal perception for all participants, averaged across the low contrast condition.

Binocular rivalry arises in spatially distinct zones (Blake et al., 1992). InFoRM enables the experimenter to estimate the spatial extend of these zones by calculating the mean blob size observers reported during their piecemeal perceptual reports. Specifically, during Phase 1) the participant learnt that the forward and backward movement of the joystick changed the sizes and numbers of patches of each rivaling stimulus within their experience (Figure 6C). We calculated distribution of blob sizes measured during piecemeal reports for each participant’s data and fitted those with Gaussian function (Figure 6D). Averaged across trials, contrasts, and participants, the mean blob size (Gaussian: mu) was 0.67° (median 0.68°).

3.5. Agreement between joystick responses generated with perceptual and physical alternations

Analysis of the cross correlation between the Phase 3) Rivalry reports and Phase 4) Replay-Me results found that the median response delays (median across trials, participants 170ms ± 19ms) were not significantly different across contrast conditions. The agreement between perceptual rivalry and the physical replay reports was 73% ± 5%, averaged across contrast conditions. We found a significant effect of stimulus contrast on agreement between perceptual Rivalry (Phase 3) and physical changes during Phase 4 ‘Replay Me’ responses F(5.3)=81, p=0.007, which was due to a significantly lower accuracy for the mixed contrast compared (median 69%) to high contrast condition 80% (low contrast= 73%) (see Supplementary Materials).

4. Discussion

InFoRM is to our knowledge the only quantitative method that individually measures and validates introspection of a participant’s conscious visual experience, while not relying on post hoc qualitative measures after an experiment (Niikawa, Miyahara, Hamada, & Nishida, 2020). The InFoRM method is the first of its kind that generates validated estimates of perceptual experiences during multistability perception, namely exclusive, piecemeal, and equal-, left-, and right-predominance superimposition generated using a binocular rivalry (Liu et al., 1992; Wheatstone, 1838; Yang, Rose, & Blake, 1992), an important paradigm used to study the locus of visual consciousness in humans (Tong, Meng, & Blake, 2006) and primates (Leopold & Logothetis, 1996). Compared to all currently used report-paradigms, InFoRM has an extremely high, continuous temporal resolution, allowing novel analysis of dynamic visual experiences between states (Figure 5) and within mixed perceptual states and (Figure 6) that cannot be generated with any other currently known method.

Although beyond the scope of the current study, by applying unsupervised machine learning classification, InFoRM enables the user also to study how many perceptual states each observer actually experiences (Skerswetat and Bex, 2022, in prep.). InFoRM rivalry leaves the participant blind to the actual moment when rivalry is being measured because the experimenter does not reveal when true rivalry is actually being measured, polarized glasses are worn throughout the experiment, and because the tasks for phase 2–4 are the same.

Classification of perceptual states maps generated from InFoRM responses are consistent with button report paradigms, and demonstrated novel, significant inter-individual differences between participants both within and between contrast conditions (Figure 2). Also, the analysis of perceptual state changes via our dynamic tracking approach revealed that perception does not alter abruptly nor at a constant gradual rate, but rather changes at varying degrees of transitional speeds between states (Figure 5) and within mixed perceptual states (Figure 6). Mutual inhibition via excitation/inhibition mechanism have been suggested to drive the dynamics of alternations between perceptual states (Blake, 1989; Tong et al., 2006) and are shown to be disrupted in neuro-atypical groups (Mentch, Spiegel, Ricciardi, & Robertson, 2019; Robertson, Ratai, & Kanwisher, 2016). Perceptual transition speed between two perceptual states has been shown to be linked to cortical activity in early visual cortex V1 ⁴⁴ and different perceptual state are thought to be processed by different neural sites (Klink et al., 2010; Knapen, Brascamp, Pearson, van Ee, & Blake, 2011; Liu et al., 1992; Lumer et al., 1998; Skerswetat et al., 2018). InFoRM is able to reveal perceptual transition speed differences between and within perceptual states, that may be directly related to underlying cortical processing.

The eye-dominance scores using bilateral low and high contrast conditions show a slight left positioning bias, possibly due to the right-handed joystick used in the current study as an inward movement was subjectively reported to be more natural than outward. In future studies, we will test this hypothesis with left hand joystick responses. Exclusive and superimposed perception may probe distinct neural correlates (Klink et al., 2010; Liu et al., 1992; Polonsky, Blake, Braun, & Heeger, 2000) and thus the eye-dominance scores in Figure 4 may be suitable to test for monocular and binocular vision disruption e.g. in patients with amblyopia or age-related macular degeneration.

Binocular rivalry exclusivity and piecemeal perception arises in spatially distinct zones (Blake et al., 1992). The estimated blob sizes had a mean size across all trials, conditions, and participants of about 0.7° diameter, which is larger than previously estimated (Blake et al., 1992). Future studies may change the stimulus parameters and image content to test the impact of those on the blob size outcomes.

In the current study, we decided to use an eight 1min-trial-per-condition approach as these values have been used in previous studies resulting in relatively long testing durations. Future studies will explore the validity of a leaner InFoRM versions that are able to collect multiple rivalry trials in around 5min.

The gravity center of the joystick was the center, thus effort was required to move the joystick in 2D space with a potential to bias the responses. InFoRM is however designed to take these biases into account as these biases would have been likely captured during the classification phase 2 “Follow Me”. Furthermore, if one would argue that a bias would occur only during phase 3 then relative proportion (Figure 3A) or eye dominance measures (Figure 4) should be systemically affected, which we do not find. Hypothetically, these biases existing could also just occur during phase 3 or phase 4, but then we would measure a larger disagreement between true and actual rivalry. Hence, we do not find that such biases systematically affected our results. InFoRM can also be used with other devices such as a computer-mouse and future studies may want to investigate the reliability between input devices further.

Hidden Markov models have been used to investigate dynamics and probabilities of perceptual changes during multistability (Naber, Gruenhage, & Einhäuser, 2010). We have recently applied a new Markovian technique for binocular rivalry (Skerswetat et al., 2022). The analysis for this approach is extensive and beyond the scope of the current paper, hence will be communicated elsewhere, however the method can be employed directly with InFoRM data. In the current study, we were using traditional binocular rivalry paradigm of static gratings, however, future studies may extend InFoRM by using complex stimuli such as faces and objects. The “Rivalry” part is purposefully detached from the “InFoRM:” part since the method can be applied to other multistable paradigms, i.e. continuous flash suppression (Koch & Crick, 2004), interocular grouping (Diaz-Caneja, 1928), or flicker-swap rivalry (Logothetis et al., 1996). Due to its validation of introspection approach, InFoRM enables users to measure estimates of introspection in neuro-atypical populations, such as people with autism since visual cognition has been shown to differ to that of neurotypical controls (Simmons et al., 2009). In these atypical populations, the assumptions of typical perceptual experiences may not be valid, so the assumption-free reports in Phase 3 coupled with the validation stage in Phase 4 may avoid experimenter preconceptions. Furthermore, the dynamics of perceptual experience may differ in atypical populations and any such differences may be captured by InFoRM and quantified with the novel measures presented in this current paper. Lastly, we used an approach in which we taught participants which perceptual states would correspond with which joystick movement during Phase 1“Indicate Me” to compare it to traditional rivalry approaches, e.g., move the joystick to the left and explore its space where the stimulus appearance corresponds to left orientated exclusivity. However, future studies may use a state-assumption-free condition, i.e., that do not provide a priori concept of expected perceptual states (e.g. “left exclusivity”) and compare the results with those gained via traditional a priori paradigm.

No report paradigms, that use eye movements (Hesse & Tsao, 2020) or pupil size changes (Naber et al., 2011; Schütz et al., 2018) have been used to make inferences of exclusive perceptual state without any active indication of perception by the observer.

InFoRM is in its current version a report-paradigm as defined in the introduction and is the first method which provides validated estimates of visual introspection, or in other words InFoRM’s novel approach is designed to estimate “what has been seen?”. However, to get an estimation of “Is something consciously seen?” no-report paradigms may also be suited. Hence, future studies will combine InFoRM with no report-approaches, i.e., first establishing the estimates of perceptual states with an input device while tracking gaze and pupil behavior and then using a view-only condition while keep tracking eyes and pupils and studying whether differential pupil and gaze behavior predicts perceptual state changes.

5. Conclusions

In conclusion, InFoRM provides personalized, validated estimates of visual introspection, and its high temporal resolution in combination with continuous estimates of perception within mixed states allow new insights into the dynamic of multistable perceptual competition. The method promises to be a novel tool for both basic science of visual consciousness and clinical research of interocular imbalance.

Highlights.

• Current binocular rivalry measures do not capture all perceptual states, have no estimates of what was seen (introspection), lack high-temporal resolution

• We introduce InFoRM (Indicate-Follow-Replay-Me) and applies it to binocular rivalry

• Temporal resolution in the current study: 3600data/min

• Measures multiple perceptual states: exclusive, piecemeal, equal-predominant, left-predominant, or right-predominant superimposition

• InFoRM provides new measures of multistable perception: introspection estimates, perceptual-state-specific dominance scores, velocity analysis for changing perceptual states, and size estimates of interocular suppression zones