Table 7:
C0 violation counts by invariant family across models on the N=200 datasets. Counts are invariant-level triggers; a single scenario may trigger multiple families. Qwen 7B fails on schema (B200) and authority (MIMICEXT200), Llama 8B on tool-use rules across both datasets, Meditron 7B on a mix of authority and tool checks, and Sonnet 4.5 and Gemma 4 have the fewest violations overall, both primarily in the tool family.
| Model | Dataset | VR (%) [95% CI] | Schema | Authority | Tool | Metamorphic |
|---|---|---|---|---|---|---|
| Qwen 7B | B200 | 65.0 [58.2, 71.3] | 88 | 38 | 24 | 3 |
| Qwen 7B | MIMICEXT200 | 69.5 [62.8, 75.5] | 33 | 105 | 63 | 3 |
| Llama 8B | B200 | 62.5 [55.6, 68.9] | 10 | 4 | 115 | 7 |
| Llama 8B | MIMICEXT200 | 72.5 [65.9, 78.2] | 1 | 37 | 139 | 1 |
| Meditron 7B | B200 | 24.0 [18.6, 30.4] | 0 | 32 | 43 | 8 |
| Meditron 7B | MIMICEXT200 | 19.5 [14.6, 25.5] | 0 | 31 | 38 | 0 |
| Sonnet 4.5 | B200 | 11.0 [7.4, 16.1] | 0 | 4 | 18 | 0 |
| Sonnet 4.5 | MIMICEXT200 | 16.0 [11.6, 21.7] | 0 | 14 | 18 | 3 |
| Gemma 4 E4B | B200 | 6.5 [3.8, 10.8] | 0 | 7 | 2 | 5 |
| Gemma 4 E4B | MIMICEXT200 | 14.0 [9.9, 19.5] | 0 | 22 | 12 | 4 |
a
VR = deploy violation rate at C0 (bare model, no sentinel). Wilson 95% CIs for N=200.
b
Schema = INV-SCHEMA-00; Authority = INV-AUTH-01/02/03; Tool = INV-TOOL-04/06; Metamorphic = INV-META-MR1/MR2/MR3/MR4.