Skip to main content

View full-text article in PMC

. 2022 Aug 17;24(8):e34705. doi: 10.2196/34705

Table 4.

Intraclass performance metrics on the test set.

Category	TF-IDF^a and SVM^b				BERT^c,d				XLNet^d
	Precision (95% CI)	Recall (95% CI)	F ₁	Precision (95% CI)		Recall (95% CI)	F ₁	Precision (95% CI)		Recall (95% CI)	F ₁
Suicidal ideation (n=57)	0.32 (21.93-43.58)	0.44 (30.74-57.64)	0.37	0.58 (43.25-73.66)		0.45 (32.36-59.34)	0.51	0.60 (46.11-74.16)		0.54 (40.66-67.64)	0.55
Coping (n=42)	0.44 (31.55-57.55)	0.64 (48.03-78.45)	0.52	0.76 (59.76-88.56)		0.69 (52.91-82.38)	0.72	0.71 (54.80-83.24)		0.74 (57.96-86.14)	0.73
Awareness (n=63)	0.65 (51.60-76.87)	0.62 (48.80-73.85)	0.63	0.71 (58.05-81.80)		0.70 (56.98-80.77)	0.70	0.69 (56.74-79.76)		0.74 (62.06-84.73)	0.72
Prevention (n=91)	0.83 (74.00-90.36)	0.82 (73.02-89.60)	0.83	0.81 (71.93-88.16)		0.89 (80.72-94.60)	0.85	0.82 (72.27-88.62)		0.87 (78.10-93.00)	0.84
Suicide cases (n=103)	0.70 (60.82-78.77)	0.74 (64.20-81.96)	0.72	0.75 (65.14-82.49)		0.77 (67.34-84.46)	0.76	0.78 (68.31-85.52)		0.75 (65.24-82.80)	0.76
Irrelevant (n=285)	0.74 (67.78-79.18)	0.63 (57.27-68.77)	0.68	0.64 (57.76-69.11)		0.65 (59.06-70.45)	0.64	0.68 (61.96-73.46)		0.64 (57.99-69.44)	0.66

^aTF-IDF: term frequency-inverse document frequency.

^bSVM: support vector machine.

^cBERT: Bidirectional Encoder Representations from Transformers.

^dScores are averages across 5 model runs for BERT and XLNet. Table S3 in Multimedia Appendix 1 shows separate runs.