Table 1 |.
Descriptions and example methods for different levels of assessments
| Category | Description | Example methods |
|---|---|---|
| Model-level assessment | Models are characterized by… | |
| Sensitivity and specificity | Examining the response patterns of predictive models to different inputs and experimental conditions | Positive and negative controls13,47,118; testing models across a range of different conditions18; construct validity84; tuning curves of brain patterns86 |
| Generalizability | Testing models on multiple datasets from different samples, contexts and populations | Research consortia and multisite collaborations119 |
| Behavioral analysis | Analyzing the patterns of model decisions and behaviors over many instances and examples (or over time for adaptive models) | Methods analogous to psychological tests50; analyses at given time points52; error analysis120 |
| Representational analysis | Analyzing model representations using examples or representational distance | Deep dream71; deep k-nearest neighbors121; representational similarity analysis56 |
| Analysis of confounds | Examining whether any confounding factors contribute to the model (e.g., head movement, physiological confounds or other nuisance variables) | Comparison of the model of the signal of interest with a model of the nuisance variables122 |
| Feature-level assessment | Significant features are identified by… | |
| Stability | Measuring the stability of the selected features and predictive weights over multiple tests (e.g., cross-validation or resampling) | Stability analysis61,123; bootstrap test13,124; surviving count on random subspaces125; pattern reproducibility126 |
| Importance | Measuring the impact of features on a prediction | rFe63,127,128; variable importance in projection61; sensitivity analysis66; feature importance ranking measure129; measure of feature importance130; leave-one-covariate-out131; LRP64,69; local model-agnostic explanations (LIME)67; deep learning feature importance132; Shapley additive explanations (SHAP)133; regularization42–44; virtual lesion analysis47; in silico node deletion134; weight-activation product135 |
| Visualization | Visualizing feature-level properties | Class model visualization136; saliency map70; weight visualization18 |
| Biology-level assessment | Neurobiological basis is established by… | |
| Literature | Relating predictive models to previous findings from literature across different tasks, modalities and species (e.g., meta-analysis) | Meta-analysis18,77; large-scale resting-state brain networks83,84 |
| Invasive studies | Using more invasive methods, such as molecular, physiological and intervention-based approaches | Gene overexpression and drug injection88; transcranial magnetic stimulation90; postmortem assay91; optogenetic fMRI89 |