Table 3.

Performance of the selected models on immune-related adverse event extraction, measured by macro-average F1-score with 95% CI. Higher values indicate better performance.

Category	Mistral-small	Phi4	Meditron3-Phi4	Falcon3	Llama3.1	Meditron3
Cardiac	0.36	0.33	0.40	0.31	0.00	0.40
Colitis	0.46	0.54	0.53	0.44	0.54	0.45
Diabetes	0.25	0.55	0.17	0.00	0.00	0.00
Hepatitis	0.38	0.36	0.33	0.29	0.18	0.09
Hypophysitis	0.31	0.32	0.31	0.12	0.12	0.17
Nephrological	0.08	0.22	0.35	0.24	0.21	0.29
Neurological	0.08	0.11	0.11	0.03	0.06	0.00
Pancreatitis	0.55	0.67	0.00	0.57	0.67	0.40
Pneumonitis	0.46	0.53	0.43	0.29	0.22	0.00
Rheumatological	0.07	0.09	0.08	0.00	0.00	0.00
Skin	0.21	0.20	0.26	0.18	0.29	0.18
Thyroiditis	0.30	0.34	0.29	0.31	0.24	0.14
Overall (95% CI)	0.29 (0.20-0.35)	0.35 (0.25-0.42)	0.26 (0.19-0.32)	0.23 (0.15-0.28)	0.21 (0.12-0.26)	0.17 (0.08-0.25)