Table 2.
Performance of the selected models on the PHIa detection task, measured by macro-average F1-score with 95% CI. Higher values indicate better performance. The column headings NAME, DATE, TIME, ID, AGE, TEL, and ADDRESS represent the original PHI entity categories.
| Model | Size | NAME | DATE | TIME | ID | AGE | TEL | ADDRESS | Overall (95% CI) |
| Mistral-small | 24B | 0.84b | 0.11 | 0.83 | 0.26 | 0.53b | 0.06 | 0.08b | 0.33 (0.32-0.35) |
| Phi4 | 14B | 0.00 | 0.77 | 0.88 | 0.51 | 0.53b | 0.06 | 0.01 | 0.34 (0.32-0.36) |
| Meditron3-Phi4 | 14B | 0.00 | 0.89 | 0.90b | 0.46b | 0.43 | 0.06 | 0.01 | 0.34 (0.32-0.36) |
| Falcon3 | 10B | 0.00 | 0.90b | 0.87 | 0.31 | 0.53 | 0.06 | 0.04 | 0.34 (0.31-0.35) |
| Llama3.1 | 8B | 0.00 | 0.49 | 0.61 | 0.16 | 0.26 | 0.01 | 0.04 | 0.20 (0.17-0.22) |
| Meditron3 | 8B | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 (0.00-0.00) |
| Fine-tuned RoBERTa | — | 0.95 | 0.99 | 1.00 | 0.81 | 0.74 | 1.00 | 0.79 | 0.94 (0.91-0.96) |
| Support, n | — | 541 | 1336 | 314 | 25 | 138 | 166 | 571 | 3091 |
aPHI: protected health information.
bHighest performance among the tested large language models for each PHI entity category.
cNot applicable.