Table 4.
Response classification and performance by scenario complexity. Complexity strata pool observations across models and prompt arms, so each scenario contributes 18 observations (3 models × 2 arms × 3 repetitions); the design-effect correction therefore uses m=18 for these estimates, and ICCsa were estimated separately for each outcome. The corresponding unsafe major error rates were 3.1% (95% CI 1.1%-8.3%) for low complexity, 9.5% (95% CI 5.3%-16.5%) for medium complexity, and 29% (95% CI 19.9%-40.1%) for high complexity.
| Complexity | Observations, n | Exact correct, n | Safe but imperfect, n | Unsafe major error, n | Unsafe major error-free rate, % (95% CI) | Exact concordance rate, % (95% CI) |
| Low | 288 | 275 | 4 | 9 | 96.9 (91.7-98.9) | 95.5 (90.1-98) |
| Medium | 378 | 297 | 45 | 36 | 90.5 (83.5-94.7) | 78.6 (69.1-85.8) |
| High | 414 | 200 | 94 | 120 | 71 (59.9-80.1) | 48.3 (39.5-57.3) |
aICC: intraclass correlation coefficient.