Table 2.
Simulation Results for the Four Edge Case Designs.
|
| |||||||
|---|---|---|---|---|---|---|---|
| DESIGN | MIRRORED STUDY DESIGN | R-R-S-COMBINATION | RANGE OF CORRELATION | CENTERED 95% QUANTILE [0.025; 0.975] | VARIANCE RANGE | NON-NORMAL | OUTLIERS |
|
| |||||||
| A | Fischer et al. (1982) | 5-6-21 | 0.18; 0.97 | 0.6; 0.93 | 0.14; 2.47 | 6.76% | 0.07% |
|
| |||||||
| B | Farina et al. (1977) | 14-2-23 | –0.21; 0.96 | 0.47; 0.9 | 0.03; 4.35 | 16% | 0.3% |
|
| |||||||
| C | Koscinski* (2013) | 10-10-40 | 0.67; 0.96 | 0.82; 0.94 | 0.28; 1.56 | 2.62% | 0.36% |
|
| |||||||
| D | Rubenstein* (2005) | 40-40-40 | 0.93; 0.99 | 0.95; 0.98 | 0.48; 1.09 | 0% | 0% |
|
| |||||||
Note: R-R-S are the number of raters in group one and group two (R-R), and the number of rated stimuli (S); Monte Carlo samples were collected with niter = 100,000 iterations; the outlier column shows the percentage of samples in which any outliers were present and the non-normal column the percentage of non-normal distributions in the samples; * the number of rated stimuli S was reduced to ns = 40 due to a constrained number of stimuli in our dataset.