Table 3.

AUROC^a and AUPRC^b on the combined MGH^c and BIDMC^d test data for the logistic regression, random forests, and RoBERTa embeddings models, using different types of model input (ICD^e codes, medications, and notes). The best performances are italicized.

Input	Logistic regression			Random forests			RoBERTa
	AUROC (95% CI)	AUPRC (95% CI)	AUROC (95% CI)		AUPRC (95% CI)	AUROC (95% CI)		AUPRC (95% CI)
ICD codes, medications, and notes	0.968 (0.940-0.982)	0.921 (0.835-0.969)	0.962 (0.719-0.991)		0.894 (0.720-0.985)	0.866 (0.821-0.904)		0.766 (0.632-0.853)
Notes only	0.964 (0.942-0.975)	0.901 (0.805-0.956)	0.963 (0.916-0.976)		0.899 (0.659-0.968)	—f		—
ICD codes only	0.646 (0.580-0.826)	0.557 (0.486-0.716)	0.646 (0.595-0.835)		0.557 (0.492-0.720)	—		—
Medications only	0.639 (0.338-0.875)	0.428 (0.235-0.651)	0.644 (0.326-0.936)		0.459 (0.320-0.779)	—		—

^aAUROC: area under the receiver operating characteristic curve.

^bAUPRC: area under the precision-recall curve.

^cMGH: Mass General Hospital.

^dBIDMC: Beth Israel Deaconess Medical Center.

^eICD: International Classification of Diseases.

^fNot applicable.