Table 6.
Intraclass correlations for between-groups designs. The table shows that the intraclass correlations mostly reached the desired level of ICC2 = .80 when multiple observations were made per condition and all the items used. In general, this improved the interpretation (see in particular the study of Pyc & Rawson, 2010). At the same time, for some dependent variable (e.g., rating scales) rather stable data can be obtained with a few questions (see the study by Wilson et al., 2014).
| Study | Dependent variable | N parts | N conditions | Nobs_per_ cond | Across the entire dataset | Average within condition | dN = 1 | dN = tot | ||
|---|---|---|---|---|---|---|---|---|---|---|
| ICC1 | ICC2 | ICC1 | ICC2 | |||||||
| Camerer et al. (2018) | ||||||||||
| Ackerman et al. (2010) | Evaluating job candicates | 599 | 2 | 8 | 0.47 | 0.87 | 0.47 | 0.88 | 0.10 | 0.13 |
| Gervais & Norenzayan (2012) | Belief in God | 531 | 2 | 1 | NA | NA | NA | NA | –0.07 | –0.07 |
| Karpicke & Blunt (2011) | Text memory | 49 | 2 | 1 | NA | NA | NA | NA | 0.83 | 0.83 |
| Kidd & Castano (2013) | Emotion recognition | 714 | 2 | 36 | 0.09 | 0.78 | 0.09 | 0.78 | –0.03 | –0.08 |
| Morewedge et al. (2010) | M&Ms eaten | 89 | 2 | 1 | NA | NA | NA | NA | 0.75 | 0.75 |
| Pyc & Rawson (2010) | Word translations | 306 | 2 | 48 | 0.14 | 0.89 | 0.14 | 0.88 | 0.12 | 0.30 |
| Shah et al (2012) | Dots-mixed task | 619 | 2 | 1 | NA | NA | NA | NA | –0.03 | –0.03 |
| Wilson et al (2014) | Enjoyment ratings | 39 | 2 | 3 | 0.82 | 0.93 | 0.73 | 0.89 | 1.32 | 1.44 |