Fig. 7.

Outcomes of the frequentist analysis for different numbers of simulated trials (K) and participants (N). Top row: t values for δ = 0 (left subplot) and δ = 1 (right subplot). Dotted lines show t = 0, dashed lines show the critical t value in a two-sided t test with α = .05, and red lines show the theoretical t value. Dots are true t values (TR; blue), t values from a hierarchical frequentist strategy (HF; green), non-hierarchical frequentist strategy (NF; grey), and two-step frequentist strategy (TF; orange); asterisks denote outliers (outliers are jittered to prevent visual overlap). Numbers at the bottom indicate the proportion of significant t values (out of 200 t tests). Bottom row: p values for δ = 0 (left subplot) and for δ = 1 (right subplot). Solid lines indicate p = .05. Dots are true p values (blue), p values from a hierarchical frequentist strategy (green), non-hierarchical strategy (grey), and two-step frequentist strategy (orange). Data points are jittered for improved visibility