Protocol for Neuro-MINE: A machine learning pipeline for time-series neural response data analysis

Summary

Here, we present a protocol for the machine-learning-based Python package Neuro-MINE. We provide instructions for user interfaces that control model training and model prediction. Model training links experimental predictors to neural activity using a convolutional neural network that captures both linear and non-linear relationships. Taylor decomposition characterizes key aspects of these relationships, and we provide instructions for tweaking relevant parameters. Model prediction then computes the most likely neural response to hypothetical inputs, making it a powerful hypothesis generator.

For complete details on the use and execution of this protocol, please refer to Costabile et al.¹

Graphical abstract

Highlights

• •Open-source pipeline to analyze neural response data
• •Identification of neurons explainable by task inputs
• •Generation of human and machine-readable insights
• •Prediction of responses to novel inputs

Publisher’s note: Undertaking any experimental protocol requires adherence to local institutional guidelines for laboratory safety and ethics.

Here, we present a protocol for the machine-learning-based Python package Neuro-MINE. We provide instructions for user interfaces that control model training and model prediction. Model training links experimental predictors to neural activity using a convolutional neural network that captures both linear and non-linear relationships. Taylor decomposition characterizes key aspects of these relationships, and we provide instructions for tweaking relevant parameters. Model prediction then computes the most likely neural response to hypothetical inputs, making it a powerful hypothesis generator.

Before you begin

Modern neuroscience methods generate vast quantities of data² by recording from many neurons simultaneously as animals perform behavioral tasks, for example, using high-density field recordings. A common question is how the activity of neurons relates to stimuli the animal experiences or behaviors the animal performs. A classical approach to this problem is linear regression³: a powerful and interpretable method to relate predictors (such as stimuli and behaviors) to neural activity and determine which predictors drive a given neural response. Linear regression can furthermore predict the activity of neurons that filter their inputs over time (such as through integration or differentiation) by computing coefficients for multiple time shifted versions of the predictors.⁴ However, by construction, linear regression is limited to capturing linear relationships between predictors and neural activity. Therefore, a more robust analysis method is needed to capture non-linear relationships, i.e., Neuro-MINE. Neuro-MINE (Figure 1) can be used to analyze any data for which it is expected that features of interest are encoded in the time domain. For neural data this includes cellular electrophysiology, calcium or voltage imaging, and fMRI data, while electroencephalogram (EEG) methods are likely not suitable, since important features are encoded in the frequency domain. This protocol outlines step-by-step instructions using a new graphical user interface (GUI) for MINE,¹ a method we previously developed that overcomes limitations of linear approaches to defining the relationship between numerical predictors and neural activity.

Figure 1.

Neuro-MINE pipeline

(A) Input acquired data.

(B) Graphical user interface.

(C) Illustration of convolutional neural network (CNN) architecture.

(D) Taylor series expansion formula.

(E) Output insights from analysis.

Innovation

To overcome limitations of linear methods for relating predictors to neural activity, we developed Model Identification of Neural Encoding (MINE).¹ Instead of linear regression, MINE uses a convolutional neural network (CNN) to fit the relationship between predictors and neural responses. Each layer in the CNN performs a nonlinear transformation using the infinitely differentiable activation function swish. This allows the model to uncover nonlinear dependencies and transformations performed by neurons across time, such as differentiation and integration,¹ which can be observed in the receptive fields computed by MINE. In the context of neural circuits, differentiation appears whenever neurons show adapting (phasic) responses, while integration is commonly found if neurons show facilitating responses or perform integration-to-bound. Importantly, MINE can accommodate any form of numerical, time-series response. However, as neuroscientists we have tailored the model to neural responses. These responses can be continuous as obtained during calcium imaging or fMRI, as well as discrete responses such as spikes.^1,5 However, a general problem of nonlinear methods is interpretability, e.g., it is generally more difficult to determine which predictors drive a given response. Taylor analysis allows the decomposition of most mathematical functions into an infinite series of polynomial terms. Importantly, while different inputs influence each other within the network, they are separable using Taylor decomposition. MINE therefore addresses the interpretability problem by approximating the network with a Taylor expansion up to the second order term, which allows the contribution of each individual input in driving the network output to be assessed, similar to evaluating the separable feature effects in a linear model. This approach defines important predictors, characterizes the nonlinearity (or linearity) of the relationship between predictors and neural activity, and allows extracting receptive fields.¹ MINE is currently limited to 1-dimensional input data features. This means it is not suitable to directly process complex movie data as input stimuli, which are fairly common in fMRI experiments. However, global features such as motion direction, contrast, coherence of grating, or dot-motion stimuli are well-suited as inputs. Therefore, as long as features of interest are extracted from the movie stimuli first, experiments using these stimuli can be processed by MINE.

While MINE is a powerful approach, the original publication lacked step-by-step instructions and ease of access. Here we describe a protocol for using Neuro-MINE (v=1.0), a self-contained Python package hosted on PyPi. Notably, Neuro-MINE has two modules, both of which are accessible via graphical user interfaces and the command line. The first module trains nonlinear models and generates insights from the models, while the second module predicts responses to novel inputs. The following protocol therefore provides an accessible approach to using nonlinear models for analyzing neural data.

Key resources table

REAGENT or RESOURCE	SOURCE	IDENTIFIER
Deposited data
Test data and dataset from: Single-trial neural dynamics are dominated by richly varied movements.	Costabile et al.1 Musall et al.4	https://doi.org/10.5281/zenodo.21851176
Software and algorithms
Python 3.12	Anaconda	https://www.anaconda.com/
Neuro-MINE package	PyPi	https://pypi.org/project/neuro-mine/
Neuro-MINE code repository	GitHub	https://github.com/matovic5/neuro_mine
Neuro-MINE documentation	Read the Docs	https://neuro-mine.readthedocs.io/
Neuro-MINE code archive	Zenodo

Step-by-step method details

Software installation

Timing: 30 min

• 1.Install a Python package manager.

Note: We recommend and give instructions for Anaconda or Miniconda for managing Python environments, which can be freely downloaded and installed from https://www.anaconda.com/download.

• 2.Create a conda environment for the installation of Neuro-MINE.

In your operating system terminal, input the following commands:

> conda create -n mine python=3.12

> conda activate mine

> pip install -U neuro_mine∼=1.0

CRITICAL: If you intend to use an existing environment, TensorFlow<2.16 must not already be installed.

Note: We recommend using the Python version that is currently compatible with the newest release of TensorFlow (at time of publishing Python version 3.12).

• 3.Optional: Test installation and familiarize yourself with Neuro-MINE using the test files fit_test_predictors.csv and fit_test_responses.csv, which constitute a toy dataset,¹ posted on Zenodo: https://doi.org/10.5281/zenodo.20310445.

Data preparation

Timing: 0–1 h

• 4.
  Separately export data for predictors and responses for input to Neuro-MINE:

  CRITICAL: All predictor and response data files need to be in a delimited file format such as comma- or tab-separated value (CSV or TSV) files (any common delimiter will work). Both predictors (Figure 2A) and responses (Figure 2B) must be organized such that each column is a feature while timepoints run along rows.

  • a.
    Separate data into different predictor and response files if acquired on different subjects or if data from the same subject was acquired in discrete episodes or trials (i.e., episodic data).

    CRITICAL: The first column in each file must indicate time, which must increase monotonically to ensure that each datapoint is assigned to a unique timepoint (Figures 2A and 2B).

    CRITICAL: For episodic data, predictor and response file names must follow the same alphabetical order, as Neuro-MINE uses alphabetical sorting to match predictor and response files across episodes.

    CRITICAL: Neuro-MINE must be able to relate timepoints in response and predictor files. This means that they must share a common format. E.g., if responses count time since experiment start, predictors cannot use clock time but must also refer to time since experiment start.

    CRITICAL: Event-type responses, such as spikes, must be encoded as a series of 0s and 1s in the response files. It is recommended that these are encoded as integers, not floating-point values, since problems in precision can lead to the misinterpretation of responses as continuous rather than spiking data. For event-type predictors, these can either be encoded as values of 0s and 1s or alternatively, they can be indicated by words (e.g., “response”, “no-response”).

    Note: While categorical predictors with more than two levels (e.g., “blue”, “red”, “yellow” or 1, 2, 3) are supported in principle, their use is not advisable since the CNNs used by Neuro-MINE implicitly assume that a value of 3 is more different from a value of 1 than a value of 2.

    CRITICAL: Header must not be numeric, otherwise Neuro-MINE cannot distinguish headers from data.

    Note: If predictors and or responses should be specifically named in the analysis files, provide a column header in the predictor and/or responses files naming the data.

    Note: If no header is present, the columns in the predictor file will be named [“Time”, “P_0”, “P_1”…] by default and in the response file [“Time”, “R_0”, “R_1” …].

• 5.
  Optional: Compute pairwise correlations of the input data (Figure 3A) and understand auto-correlation timescales for predictors and responses (Figure 3B).

  Note: For data curation and pre-analysis, it is useful to compute input data pairwise correlations (Figure 3A) and auto-correlation timescales for predictors and responses (Figure 3B). This will help setting model-history parameters and downsampling (see parameter tuning).

  • a.
    Autocorrelations and correlations can be generated in the command line interface.
    In your operating system terminal, input the following commands:

    > conda activate mine

    > Data-diagnostic -p <predictor files> -r <response files> -ut <keep if using time> -eps <keep if data is episodic> -o <output directory>

    Note: As in linear regression, it is advantageous to remove pairwise correlation dependencies across predictors (Figure 3A) through orthogonalization.

    Note: As with all neural networks, the more data that is provided to the model, the better the training results will be. The precise amount of data required depends on the quality (e.g., amount of noise, how nonlinear the relationships are) and if the signal-to-noise ratio is high. Even a few hundred timepoints can produce useable insights.¹

Figure 2.

Examples of input data file structure

(A) Excerpt of example predictor file structure; data from Musall et al., (2019).⁴

(B) Excerpt of example response file structure; data from Musall et al., (2019).⁴

Figure 3.

Data diagnostics

(A) Predictor pairwise correlations.

(B) Autocorrelation times for predictors and responses.

Model training

Timing: 10 min to 1 day

This step is the key function of Neuro-MINE. During training, for each response, a CNN will be fit to explain the response in terms of the predictors. For all responses where the quality of the fit passes a user-set threshold, additional metrics will be computed. These include which predictors contribute to the response, whether the response is linearly or nonlinearly related to the predictors, and optionally receptive fields will be extracted.

Note: Run time will scale with response count, the number of datapoints and the number of training epochs. Run time will also increase with increasing numbers of predictors and increasing history length (also see troubleshooting, problem 3).

• 6.Activate the environment where the package neuro-mine is installed by running this command in your terminal:

> conda activate mine

• 7.Launch Training graphical user interface (Figure 4A) by running this command:

> Mine

• 8.Choose an “Output Directory”, which will be the folder all the output files will be written to.

Note: If the output directory does not exist, a new one will be written.

• 9.
  Select “Browse Predictor File(s)…” to select any number of CSV/TSV files.

  • a.
    If selecting many files, hold down the control/command key to multi-select.
• 10.
  Select “Browse Response File(s)…” to select any number of CSV/TSV files.

  • a.
    If selecting many files, hold down the control/command key to multi-select.

Note: Neuro-MINE is equipped to process episodic data where episodes are spread across separate files.

• 11.
  Use the default Processing Parameters; to change the Processing Parameters, consider the following:

  • a.
    Select “Use Time as a Predictor” if the time column in the predictor file should itself be used as a predictor.

    Note: This can be useful in cases where signals change across time because of indicator bleaching.

  • b.
    Select “Data is Episodic” if all selected predictor and response files belong to separate episodes of the same experiment (i.e., if the same test subject had multiple, non-continuous trials tested on them).

    Note: Episodic processing will take care not to generate training samples or analyses across episode boundaries making it the preferred method for analyzing such data rather than combining all episodes into one file.

  • c.
    Select “Downsample Data” and input a “Downsampling Factor” > 1 to reduce the sample rate in the predictor and response files through averaging.

    Note: Setting the downsampling factor to 4 will reduce the dataset size to a quarter (i.e., it divides it by a factor of 4).

    Note: Continuous data is averaged and the last timepoint is selected so that no future is included in the data. E.g., if a factor of 4 is input, every 4 rows, the predictor and response values will be averaged, and the 4^th time point will be used to represent the data.

    Note: For event data (spike responses and 0–1 encoded predictors), the maximum value is chosen. Thus, if at least one event happened in the downsampling window, a 1 is assigned, otherwise a 0.

    Note: Downsampling increases training and analysis speed and can improve fits by reducing noise. It is worthwhile to consider which final sampling rate is really needed to obtain useful results. E.g., for a calcium indicator with 2 s decay time, response data that is sampled at 30 Hz is likely oversampled.

• 12.
  Use the default Fitting Parameters; to change the Fitting Parameters, consider the following:

  • a.
    “Test Score Threshold”:

    • i.
      Input the minimal correlation between model predictions and true outcomes needed on test data to consider a response “fit”.
    • ii.
      Changing this value will have the greatest influence on results because it filters responses whose test correlation is below the threshold.
    • iii.
      By default, it is set to the square root of 0.5 (√0.5) indicating that the prediction explains 50% of the variance in the data. Empirically, this threshold yielded a >90-fold enrichment of true over false positives. However, refer to parameter tuning, to set a threshold based on the data being analyzed.
  • b.
    “Model History (sec)”:

    • i.
      Number of seconds of past data used for fitting (longer history increases runtime). Set this based on expectations about how far in the past events might influence current neural activity.

      Note: Inputs are currently limited to past events. For behaviors, neurons may encode anticipatory activity which would require future inputs (negative history) for detection. While not supported directly, a similar effect can be achieved by time-shifting predictors relative to responses (Costabile et al., 2023).¹

  • c.
    “Number of Epochs”:

    • i.
      Number of iterations over the entire training dataset used to fit the model.
    • ii.
      The default value is 100 but users are encouraged to select “Training Curves” in the GUI to generate curves of fitting score vs epochs to tune this parameter (see Step 17.b. and parameter tuning).
  • d.
    “Train Data Fraction”:

    • i.
      Fraction of data used for training (the remainder is used for testing).

      Note: To test generalization, the values of predictors in the test period should differ from those in the training period. In other words, it is advisable to plan for the test period when designing experiments, ensuring that somewhat different stimulus values or different temporal sequences of stimulus values are presented. This is less problematic for behavioral predictors, as the animal is unlikely to exactly replicate its behavior.

      Note: For episodic data, train/test sets are split by episodes. For example, if an experiment contains 10 episodes, the first 8 are used for training and the last 2 for testing at the default value of “Train Data Fraction”.

• 13.
  Use the default Taylor Expansion Parameters; if changing parameter values, consider the following:

  • a.
    “Taylor Cutoff”:

    • i.
      Minimal fraction of variance explained that must be lost when a predictor is removed for it to be considered as driving the response.
    • ii.
      A value of 0.1 is a sensible default when neurons respond robustly. If neurons are expected to only respond to a stimulus occasionally, a value of 0 may be more appropriate.

      Note: This parameter determines how important individual predictors must be for making the determination that they influence the response. The default value has been empirically determined to be robust in Costabile et al. (2023).¹ However, in cases where there are many predictors that are linearly related (as in the example data in Figure 4B) the default cutoff might result in most responses having no assigned predictors. As in linear regression it is therefore advisable to orthogonalize predictors and if there is a large number of predictors (>5) it is worthwhile to tune the parameter on a validation dataset that is separate from the final training and test datasets (see parameter tuning).

  • b.
    “Significance Threshold”:

    Note: After correcting for multiple comparisons, the loss in explained variance must be significantly larger than the Cutoff given this p-value threshold for a predictor to be considered to drive a response (note that a Bonferroni correction is used to correct p-values for multiple comparisons across responses).

  • c.
    “Linear Fit Variance Fraction”:

    • i.
      Threshold on the fraction of variance explained by a linear expansion of the model. If surpassed, the neural response is classified as linear.
  • d.
    “Square Fit Variance Fraction”:

    • i.
      Threshold on the fraction of variance explained by a second-order expansion.

      Note: If surpassed (and the linear threshold is not), the response is classified as second (i.e., square) order. If neither threshold is crossed, it indicates that the response is higher than second order, i.e., cubic or greater.

• 14.Optional: Select “Restore Presets” button to revert to default Processing and Fitting Parameters, Run Diagnostics, and Additional Outputs.
• 15.Optional: Select “Save Parameters” button to generate and save a JSON file with current configuration to be loaded or referenced at a later time.
• 16.Optional: Select “Populate Parameters from JSON…” button to load JSON file from a previous fitting trial or parameters that had been previously saved.
• 17.
  Use default Generate Diagnostics settings; if changing the setting, consider the following:

  • a.
    “False Discovery Rates”:

    • i.
      Response data is circularly permuted, and models are refit to determine validity of test score threshold on non-permuted results.
    • ii.
      False discovery rates are calculated for each Test Score Threshold by dividing Permuted Data score with Test Data score, which is saved as a plot (see Step 24 for interpretation).

      Note: Users are strongly encouraged to select this option to gain insights about data quality from test metrics, especially if analyzing datatypes that have not been previously validated.

  • b.
    “Training Curves”:

    • i.
      Training and test scores are calculated across epochs and saved as a plot (see Step 25 for interpretation).

      Note: These training curves can be used to judge the required amount of training epochs and whether overfitting is occurring (see parameter tuning).

• 18.
  Use default Additional Outputs; if changing the setting, consider the following:

  • a.
    “Store Linear Receptive Fields (Jacobians)”:

    • i.
      Linear receptive fields for each fit neuron and predictor will be saved as a CSV file (see Step 26 for interpretation).
• 19.
  Start model training:

  • a.
    Use default setting for “Force Run with Memory Warning”; if changing the setting, consider the following:

    • i.
      Check this button to force the model fitting to run despite anticipated excessive memory usage.

      Note: Depending on the computational memory and data size, running the model might significantly decrease system performance.

  • b.
    Start model training by selecting “Run Model” button:

    CRITICAL: This button can be selected only if the Output Directory and Predictor/Response File(s) have been selected and the numerical fields are correctly filled. That is, the “Downsampling Factor”, Fitting Parameters, and Taylor Expansion Parameters are numerically constrained in the GUI to values specified next to the field. The GUI will not accept values outside the predefined constraints, which will be indicated by the entry field turning red. The “Run Model” button will not be functional until the constrained values have been correctly input.

Figure 4.

Neuro-MINE Train user interface

(A) Annotated version of the Neuro-MINE train GUI. S# refers to the steps in the protocol that discuss the indicated parameter.

(B) Example of command line output with updates provided during fitting.

Result interpretation

Timing: 30 min

Neuro-MINE produces graphical overviews of major data-features as well as a human-readable insight file summarizing key results (Figure 5) (see expected outcomes). Receptive field result files will be generated as well, if selected during training (see Step 18.a). For more advanced analyses, all result data is provided in HDF5 format and individual model weights are also stored. All Neuro-MINE classes and functions are importable in user-written scripts which can be used to instantiate trained models in custom code (not described).

• 20.All files resulting from the fit are named <Response File Name_…>.
• 21.
  Examine < Response File Name >_Insights.csv (Figure 5A):

  • a.
    This file combines critical information about the fit of each response.
  • b.
    The first column lists the provided or automatically assigned response names.
  • c.
    The second column indicates whether the test correlation of the trained model passed threshold (“Y”) or not (“N”).
  • d.
    The following columns (as many as predictors used in the training) list for each predictor whether it contributed to the response (“Y”) or not (“N”).
  • e.
    The second-to last column lists the determination of the linearity of the response, specifically whether it was “linear”, had a 2^nd order dependency on the inputs “square”, or if it was 3^rd and higher-order (“cubic+”). For detailed description of linearity and nonlinearity definitions refer to “Characterizing the linearity and complexity of the transformation” in Costabile et al. (2023).¹
  • f.
    The last column provides the barcode cluster number for each neuron that was successfully fit.

    • i.
      All neurons with the same number respond to the same group of predictors, i.e., they are grouped on the UpSet-Plot (see Step 22).
    • ii.
      The barcode cluster numbers are the decimal representation of the binary indicator of group assignment.

Note: The insight file can also be used to obtain quick statistics on the data either directly in a spreadsheet program or by importing the data into a pandas DataFrame within Python.

• 22.
  Examine < Response File Name >_BarcodeUpsetPlot.pdf (Figure 5B):

  • a.
    The UpSet-Plot visualizes clustering of neurons based on which predictors (rows in the plot) they respond to and whether the responses are considered linear or nonlinear according to the definition indicated in Steps 13c and 13d and explained in Costabile et al. (2023).¹
  • b.
    Each column in the plot lists one specific response cluster. The bar on top of the column shows the count of neurons within the cluster, while the dots visualize which predictors contribute above the set threshold to the responses in the cluster.
  • c.
    The bars to the left of the dot-plot quantify how many neurons in total respond to each predictor and indicate how many are nonlinear (first row).
• 23.
  Examine <Response File Name>_LinearityMetrics.pdf (Figure 5C):

  • a.
    The plot shows fit neurons with their computed “Linear Fit Variance Fraction” and “Square Fit Variance Fraction” as the x and y coordinates, respectively.
  • b.
    The dashed lines represent the threshold parameters set before training.
• 24.
  Optional: Examine <Response File Name>_TestMetrics.pdf (Figure 5D), if “False Discovery Rates” was selected:

  • a.
    The top-plot in the file shows the fraction of responses that would be considered fit across a range of test-correlation thresholds for both the real and the circularly permuted data.
  • b.
    The bottom plot quantifies the false discovery rate of true over permuted data for each test correlation threshold.
• 25.
  Optional: Examine <Response File Name>_TrainCurve.pdf (Figure 5E), if “Train Curve” was selected:

  • a.
    Shows the test and training correlation scores across epochs. The test correlation score is the same score that is used to judge whether a response was fit by Neuro-MINE.
• 26.
  Optional: Examine <Response File Name>_ReceptiveFields_<Predictor name>.csv, if “Store Linear Receptive Fields (Jacobians)” was selected:

  • a.
    Receptive fields are grouped into separate CSV files for each predictor.
  • b.
    Each file contains a header row that indicates the history time of each frame in second.
  • c.
    Each row contains one response that passed the user-set test correlation threshold.

Note: It is unlikely that all neurons will share the same receptive field. Data in these files therefore needs to be aggregated in a meaningful way for analysis. As a first pass, the barcode cluster numbers in the insight file can be used. However, clustering the receptive fields themselves according to similarity as in Costabile et al. (2023),¹ is a more robust approach. To this end, any clustering method (k-means, spectral clustering, clustering on correlational similarity, etc.) can be used for grouping receptive fields.

• 27.Optional: Load <Response File Name>_analysis.hdf5 file into other user-generated analysis pipelines.

Note: As an HDF5 file these data is platform agnostic. HDF5 file libraries are available for all major programming languages. For Python we recommend h5py, which is installed alongside Neuro-MINE.

Figure 5.

Examples of generated insights from Neuro-MINE Training module on data from Musall et al., (2019)⁴

(A) Initial rows of insight file generated during fit listing each response, whether it was fit, which predictors contributed to the response, whether the relationship between the responses and predictors was linear or nonlinear and to which cluster on the UpSet-Plot (Figure 4B) the response belongs.

(B) UpSet-Plot visualizing different identified response types. Each column on the plot is one response type or cluster. Vertical bars indicate the number of responses that belong to each type. Dots show which predictors contribute to the response and whether the relationship was nonlinear or not. Horizontal bars indicate the total number of responses that are positive for a given type.

(C) Scatter plot of linearity scores for the example fits. Each dot represents one response and the value on the x-axis indicates the linear approximation score (the larger the more likely the response is linear) the y-axis indicates the squared approximation score (the larger the score the more likely the response can be explained by a 2^nd order model). The dashed lines indicate the cutoffs selected during training that are used to make the linearity determination in the insight file.

(D) Test correlation fit diagnostic plots. Top shows the fraction of responses for which the test correlation is above threshold (y-axis) for different test correlations threshold (x-axis) for the real data (blue) and circularly permuted data (orange). Black dashed line indicates user-chosen correlation threshold. Bottom shows the enrichment of real over permuted data for test correlation cutoffs. Black dashed line indicates user-chosen correlation threshold.

(E) Training curve plot. Test and train curves represent the average correlation between predictions and real data (y-axis) on the training and test portion of the data across increasing number of epochs (x-axis). This is the same metric that is used as the fit cutoff.

Parameter tuning (optional)

Timing: 1–5 h

The “Test Score Threshold” and “Taylor Cutoff” have a strong influence on analysis results. The test score threshold may need to be reduced for noisy data or data in which individual responses combine activity across many neurons (e.g., widefield calcium imaging, fMRI) while the Taylor Cutoff may have to be reduced for data with a large number of predictors (>5) or predictors that are linearly dependent.

CRITICAL: All parameter tuning should be performed on a separate validation set. This is critical to avoid biasing test results (e.g., determinations of fit or predictor contribution). We suggest using a separate set of recorded responses and/or subjects specifically for the purpose of parameter tuning. Alternatively, a specific slice of time can be used as well. Generating the validation set requires manual separation of data into distinct CSV files as Neuro-MINE currently does not provide a way to automatically split data into train, validation, and test sets.

• 28.Examine predictor and response auto-correlations (Figure 3B) (see Step 5.b.) to determine if downsampling is indicated.

Note: Auto-correlation times will vary across predictors and responses. Autocorrelation times indicate that responses at the current time are related to previous responses. Common causes for this are slow indicators (e.g., calcium indicators, BOLD responses). At the same time, slow variations in predictors can enforce auto-correlations in responses to those predictors. The plots of autocorrelation times can therefore also be used to set appropriate downsampling factors (see Step 11.c). As a rough guideline, we suggest setting the downsampling factor to the number of “Autocorrelation timepoints”, first y-axis in the plot, reported for the lower quartile of the responses. This can both save computation time and improve fits through averaging.

Note: These changes should be performed before tuning other parameters.

• 29.Start the training GUI.

> MINE

• 30.Select “False Discovery Rates” and “Training curves” and set all other parameters as desired.
• 31.Run model training on a validation subset of the data (see critical above).

Note: The larger the subset the more accurate tuning will be, especially for the test score threshold.

• 32.Examine the training curves (see Steps 17.b and 25) to gain information on the required number of epochs for model training.

Note: Drops in the test correlation may indicate over-training of the models, especially if the training correlation continues to increase. In that case, it is advisable to perform fewer training epochs and stop training before the observed drop.

Note: If the main goal of the analysis is to determine which responses can be explained based on the predictors, training can stop at the point where increases in the test correlation start to level off. However, assignments to specific predictors and the nonlinearity metrics can benefit from larger epoch counts as long as these do not lead to over-training (see above).

CRITICAL: Changes to the number of epochs should be made before tuning the following parameters.

• 33.Use the test metrics plot (see result interpretation) to determine a Test Score Threshold cutoff in which the enrichment of positives on the real data over positives on the permuted data is deemed acceptable.

Note: This might require a tradeoff between enrichment and the number of responses passing threshold as downstream analyses will only be performed on responses that cross the threshold.

Note: A false-discovery rate of 0.05 might be considered standard statistical practice, however, the cutoff ultimately has to be determined by the user.

• 34.Examine the UpSet-Plot (see result interpretation).

Note: If most responses are assigned to a category without predictor (e.g., only a dot at “nonlinear” on the UpSet-Plot), the Taylor Cutoff should be reduced (Figure 6A). However, if most predictor combinations only have a single response assigned, the cutoff may have been too lenient and should be increased (Figure 6B).

Note: The <Model-name>_analysis.hdf5 file contains the raw Taylor scores (see result interpretation), which, together with clustering in user code, can be used to determine an appropriate threshold without having to rerun the training.

Figure 6.

Parameter tuning example

(A) Example UpSet-Plot when the Taylor Cutoff is set too high - almost no data is assigned to a predictor in the dataset (first column).

(B) Part of an example UpSet-Plot when the Taylor cutoff is set too low. The majority of responses is assigned to multiple predictors leading to splits into barcode clusters, most of which have only one response each.

Model prediction (optional)

Timing: 5–60 min

A powerful feature of Neuro-MINE is that it fits nonlinear models that generalize to new inputs. The prediction step exploits this to predict each fitted response to a new set of inputs. This approach can be very useful if data was acquired in freely behaving animals in which stimuli are not under the control of the experimenter. In this case, predicting the most likely responses to a standard set of inputs can be used to perform clustering of neurons that have never been recorded under similar conditions. Predicted responses are reported on the same scale as the responses that were originally fitted. In other words, model-internal normalization used during training will be removed before saving the responses.

• 35.Activate the environment where the package neuro_mine is installed:

> conda activate mine

• 36.Launch prediction graphical user interface (Figure 7A):

> Mine-predict

• 37.Select an HDF5 Weights file from a previous model training.
• 38.Select an HDF5 Analysis file from the same training as the Weights file.
• 39.Select CSV/TSV Predictor File(s):

CRITICAL: This predictor file must have the same number of predictors as the original predictor file used during fitting. Like the predictor files used in the training module, the first column must be time (monotonically increasing). However, the timepoints do not need to be identical to the ones used during training.

Note: Prediction data will be interpolated to the same time-step size as the data used during training.

• 40.Select the JSON file generated in the same model training as the analysis and weights file.
• 41.
  Use the default Test Threshold Cutoff of −1 to perform predictions for all responses that were present during training.

  • a.
    Increase the cutoff to filter out responses with test correlations lower than the set threshold; i.e., if set to the same value as used during training, predictions will only be made for those responses that were considered fit after training.

Note: Only predictions for responses that passed the test threshold should be considered for analysis, as the test correlation is a direct measure of how well the model generalizes. However, generating predictions for non-fit responses can be useful as control data.

• 42.Select “Predict Responses” to start process, which can be monitored in the command line interface (Figure 7B) for progress.
• 43.Examine <Predictor File Name>_prediction.csv file (Figure 7C) that contains predicted responses.

Figure 7.

Neuro-MINE Predict user interface and generated result

(A) Annotated version of the Neuro-MINE prediction GUI. S# refers to the steps in the protocol which discuss the indicated parameter.

(B) Visual example of command line output with updates provided during prediction.

(C) Excerpt of predicted responses.

Expected outcomes

The training module of Neuro-MINE a) determines which neural activity can be explained by the predictors (passes the test correlation threshold), b) which of the predictors drive the responses (influenced by the Taylor cutoff), and c) what the nature of the receptive fields of these neurons are (if selected). Key insights about the model fits are provided as user-readable summaries (Figure 5), which provide information about correlations of predictions to held-out test data as an indicator of model generalization. Neurons are clustered based on fit linearity and predictor contribution, which is summarized in the main insight CSV file (Figure 5A) and an overview of the clustering is provided as an UpSet-plot (Figure 5B). Furthermore, information about the linearity of the fit (Figure 5C) is provided within the insight file as well, together with crucial information about which predictors likely drove the responses. Additionally, users can choose to repeat fitting on circularly permuted data which will provide a diagnostic plot (Figure 5D) that can be used to gauge validity of test-correlation cutoffs. Lastly, if selected, the model training loss score will be plotted with respect to epochs (Figure 5E). These insights provide a starting point to disentangle complex data sets of animal poses, locations, experienced stimuli, and neural activity regardless of the model organism used.

While the plots and insights file emphasize human readability for initial data exploration, Neuro-MINE also provides detailed outputs in portable HDF5 files. An analysis file compiles all information obtained during training including train and test correlations, raw Taylor analysis scores, linearity scores, and receptive fields (Jacobians). A second HDF5 file contains the weights of all trained models. As the Neuro-MINE package allows importing the TensorFlow model classes, these can be used for further prediction and analysis in user code using the trained CNN models.

In addition to providing critical insight about the nature of the fit, a CNN that generalizes to novel inputs for models with high test correlations offers another advantage. Namely, it allows predicting how the neurons would have responded under novel stimulus conditions.^1,5 This approach can quickly generate testable hypotheses and allow clustering activity across neurons that have not been recorded under identical conditions.

Limitations

While the goal of Neuro-MINE is to be broadly applicable to neural and other time-series data, there are some limitations in its utility. Some of these are by design while others will be addressed in future releases.

Neuro-MINE currently supports both continuous and binary-coded (spike, event) output data. Output data corresponding to counts (such as photon-counted data) can be used but the data will be interpreted as continuous without statistical benefits afforded by count data. Support for count-data with appropriate training loss functions might be added in a future release.

Neuro-MINE does not support categorical predictors. All predictors are assumed to be continuous. If categorical predictors are part of the experimental design (e.g., an animal either performs behavior A, or behavior B, or behavior C) these should be 1-hot coded. In other words, three predictors A, B, and C should be provided that are 0 everywhere except in the instance where the respective behavior was performed. This limitation is by design and likely will not be addressed.

Neuro-MINE does not support joint encoding of predictors in groups of neurons, since each individual CNN tries approximating the activity of one unit. This limitation can be partially overcome by providing principal components or other descriptions of joint activity as responses.

Neuro-MINE currently does not support dependencies of neural activity on future states of predictors. While this is sensible for sensory driven activity (as the future rarely influences the past) this might miss preparatory activity before motor actions. Support for future states as inputs will be added in a future release. Currently, this limitation can be overcome by time-shifting the predictor data relative to the response data. E.g., adding 2.5 s to all timepoints in the time column of the response data will add 2.5 s of future predictors into the response prediction (also see Costabile et al. (2023)¹).

Both the GUI and the command line interface to Neuro-MINE currently fix important model hyperparameters. These include the rate of drop-out, the l2-regulariation, and the training update rate. The values for these hyperparameters have been set to reasonable defaults that appeared to work across a range of datasets. However, it may be necessary to tune these parameters for the model to work effectively on different datasets. While this ability might be integrated into the GUI in future releases, it is already available in user code. Importing the Mine class in Python code allows changing these hyperparameters before fitting. This allows generating code to tune these parameters which is beyond the scope of the protocol but explained in the online documentation.

Troubleshooting

Problem 1

Insufficient RAM can cause the program to stop running. This only occurs in cases with very large numbers of timepoints together with long history lengths as the memory requirements scale with the product of the two. If memory errors are anticipated, an informational message will be displayed in the terminal command line (see example below) and the run will be terminated.

#### RUN STARTED ####

Responses are assumed to be continuous values not spikes

############################

The total system memory is 8 GB RAM

Expected training data size is larger than 8066 GB.

This either leads to out-of-memory errors, unexplained crashes or sluggish performance.

Reducing history length or downsampling data will reduce training data size.

Setting the downsampling factor to 64 will reduce datasize below 6 GB.

############################

### EXITING PROGRAM ###

### Either reduce memory by downsampling or set 'Force Run with Memory Warning' in GUI/pass -imw flag on command line, which will force the run to continue.

If multiple files were chosen as inputs, processing of other files will continue.

#### RUN ENDED ####

Potential solution

As per the informational message in the command line interface, the user can choose to:

• •Downsample their data by the suggested factor to reduce memory requirements. If the downsampling is within the autocorrelation time (see parameter tuning), data fidelity will not be lost.
• •Force the run and ignore the memory warning, which would strain overall system performance but might still be desirable in cases where the expected memory consumption is close to the amount of memory available.

Problem 2

Receiving an error message because there are repeating values in the time column and/or time decreases in at least one row.

ValueError: First column in datafile should be time, but values aren't strictly increasing

Potential solution

• •Inspect the values in the time column(s) and ensure that the requirements for data preparation outlined in Step 4 are met.

Problem 3

The model training runtime, which depends on the number of datapoints, the input history, epoch count, and CPU (Figure 8), exceeds reasonable analysis runtime.

Figure 8.

Heatmaps of example runtimes

(A) Processing time (in seconds) for the response that was fit for the indicated number of Datapoints (rows), History Length (columns), predictors, and epochs on a Windows 10, 3.5 GHz 10-core Intel Core i9–9900X machine.

Potential solution

• •The most effective tool for reducing runtime is to downsample the data. Downsampling is especially useful when data are oversampled. This can occur if responses are acquired at rates that exceed timescales of activity indicators or if all predictors vary on long timescales.
• •To this end it is useful to check autocorrelations of predictors and responses to estimate which reduction in sampling rate will not change the outcome.
• •If the number of datapoints is very large it is usually possible to reduce the number of training epochs without compromising the fit. This can be assessed by generating training curves (see Steps 17.b., 25, and 32).
• •The time taken for the Taylor expansion in the analysis scales with the square of the number of predictors, as Taylor scores are computed both for predictors and their interactions. It is therefore useful to limit the number of predictors to independent task variables.

Problem 4

There is no update in the command line interface when training is instantiated by selecting “Run Model” in the GUI. Windows OS does not support receiving strings longer than 8,191 characters from the GUI. If there is a large amount of predictor/response files, then the code cannot be run from the GUI.

Potential solution

• •The code can still be executed from the command line interface (described in Read the Docs: https://neuro-mine.readthedocs.io/), by pointing the analysis to a predictor folder and a response folder thereby circumventing file count limits that arise from the limited length of commands.

Problem 5

No responses pass the fit threshold.

Potential solution

This might occur because:

• •Parameter tuning is required (see parameter tuning, especially on test correlation thresholds).
• •The wrong history length has been chosen (e.g., neural responses depend on events five seconds in the past but a history length of one second was chosen). In this case it is worthwhile to increase history length, possibly on a subset of the data.
• •The data is inherently not suitable for interpretation with Neuro-MINE’s architecture. An example of data that would not yield useful insights are EEG responses. While EEG signals are dependent on input predictors, they are analyzed in the frequency and not the time domain. Given the main advantage of Neuro-MINE is the analysis of time-series datasets, EEG recordings would not result in significant fits.
• •One reason this might occur is that there is no relationship between the predictors and the responses.

Problem 6

Receiving warnings from TensorFlow when training is instantiated by selecting “Run Model” in the GUI. These warnings might look like the following:

WARNING: All 10g messages before absl:: InitializeLog() is called are written to STDERR

I0000 00:00:1785868769.744204 24320 port. cc: 153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable "TF_ENABLE_ONEDNN_OPTS=6"

WARNING: All log messages before absl:: Initializelog() is called are written to STDERR

I0000 00:00:1785868772.846340 24320 port. cc: 153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable "TF_ENABLE_ONEDNN_OPTS=0"

WARNING: All log messages before absl: :InitializeLog() is called are written to STDERR

10000 00:00:1785868777.444391

352 port. cc: 153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable "TF_ENABLE_ONEDNN_OPTS=0"

WARNING: All log messages before absl: :InitializeLog() is called are written to STDERR

I0000 00:00:1785868780.708900

352 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environ ment variable "TF_ENABLE_ONEDNN_OPTS=0".

Potential solution

These warnings are caused by problems in how TensorFlow addressed logging libraries and cannot be removed. In general, warnings written to the console do not indicate problems with fitting and can be safely ignored.

Resource availability

Lead contact

Further information and requests for resources should be directed to and will be fulfilled by the lead contact, Martin Haesemeyer (haesemeyer.1@osu.edu).

Technical contact

Problems and bugs should be directed to and will be fulfilled by the technical contact, Danica Matovic (matovic.5@osu.edu).

Materials availability

No materials were generated as part of this study.

Data and code availability

• •The software referenced in this study is freely available from https://github.com/matovic5/neuro_mine.
• •The version of record for this protocol is archived at Zenodo: https://doi.org/10.5281/zenodo.23105994.
• •Data for analysis and figures in the paper are available on Zenodo at https://doi.org/10.5281/zenodo.20310445.

Acknowledgments

Research reported in this publication was supported by the National Institute of Neurological Disorders and Stroke of the National Institutes of Health under award no. R01NS123887 to M.H. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. All authors received salary from the funders. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Author contributions

Conceptualization, D.M. and M.H.; methodology, D.M. and M.H.; data curation, M.H.; funding acquisition, M.H.; software, D.M. and M.H.; supervision, M.H.; writing – original draft creation, D.M.; writing – review and editing, D.M. and M.H.

Declaration of interests

The authors declare no competing interests.