Figure 4. Reliability for individual differences research.

(a) Test-retest reliability. The same subjects are tested with the same stimuli in the same scanner, at an appropriate time interval given the function of interest. (b) Inter-rater reliability. The same data is preprocessed in slightly different ways which are thought to be interchangeable (e.g., using physiological regressors or ICA to remove physiological noise). (c) Parallel forms reliability. The same subjects view slightly different stimuli which are thought to involve the same brain processing. (d) Out-of-sample reliability. A different set of subjects undergo the same experiment in the same scanner, and the data is analyzed in the same way. (e) Putting it all together for individual differences research. A result (e.g. relationship between fMRI-derived statistic and neuropsychological score) should be reliably obtained for different subjects at different sites, possibly using slightly different stimuli and preprocessing steps.