Table 2.
Response classification and performance by model and prompt arm. Each model-arm cell comprises 180 responses (60 scenarios × 3 repetitions). Arm 1 is the baseline clinical prompt; arm 2 is the guideline-directed prompt package. CIs are adjusted for within-scenario clustering using a design-effect correction: design effect = 1 + (m − 1) × ICC, m=3, with Wilson intervals computed on the effective sample size. In the GPT-5.3 arm 2 cell, all responses fell in the same category, so the ICCa was undefined and the number of independent scenarios (n=60) was used as the effective sample size. Unadjusted response-level Wilson intervals are provided in Table S3 in Multimedia Appendix 5.
| Model | Arm | Exact correct, n | Safe but imperfect, n | Unsafe major error, n | Unsafe major error-free rate, % (95% CI) | Exact concordance rate, % (95% CI) |
| GPT-5.3 | 1 | 156 | 16 | 8 | 95.6 (87.6-98.5) | 86.7 (75.8-93.1) |
| GPT-5.3 | 2 | 177 | 3 | 0 | 100 (94-100) | 98.3 (95.2-99.4) |
| Gemini 3 Flash | 1 | 104 | 40 | 36 | 80 (69-87.8) | 57.8 (45.3-69.4) |
| Gemini 3 Flash | 2 | 169 | 9 | 2 | 98.9 (94-99.8) | 93.9 (85.6-97.5) |
| DeepSeek V3.2 | 1 | 68 | 38 | 74 | 58.9 (46.9-69.9) | 37.8 (26.7-50.3) |
| DeepSeek V3.2 | 2 | 98 | 37 | 45 | 75 (63.2-84) | 54.4 (42-66.3) |
aICC: intraclass correlation coefficient.