Table 5.
Performance by model and complexity stratum. Each scenario contributes 6 observations (2 arms × 3 repetitions), so m=6 for those estimates. In 3 cells, all responses fell in the same category and the ICCa was undefined; for those cells, the number of independent scenarios was used as the effective sample size, as in Table 2. This panel is descriptive and was not used for formal inference.
| Model and complexity | Observations, n | Unsafe major errors, n (%) | Unsafe major error-free rate, % (95% CI) | |||
| GPT-5.3 | ||||||
|
|
Low | 96 | 0 (0) | 100 (80.6-100) | ||
|
|
Medium | 126 | 0 (0) | 100 (84.5-100) | ||
|
|
High | 138 | 8 (5.8) | 94.2 (84.4-98) | ||
| Gemini 3 Flash | ||||||
|
|
Low | 96 | 0 (0) | 100 (80.6-100) | ||
|
|
Medium | 126 | 6 (4.8) | 95.2 (86.3-98.5) | ||
|
|
High | 138 | 32 (23.2) | 76.8 (64.7-85.7) | ||
| DeepSeek V3.2 | ||||||
|
|
Low | 96 | 9 (9.4) | 90.6 (76.5-96.6) | ||
|
|
Medium | 126 | 30 (23.8) | 76.2 (61.7-86.4) | ||
|
|
High | 138 | 80 (58) | 42 (25.8-60.2) | ||
aICC: intraclass correlation coefficient.