Skip to main content

View full-text article in PMC

. 2021 Nov 3;23(11):e26777. doi: 10.2196/26777

Table 3.

Performance of natural language processing/machine learning models for fatigue domain by 3 symptom attributes.

Attributes and models		Precision (95% CI)	Sensitivity (95% CI)	Specificity (95% CI)	Accuracy (95% CI)	F1 (95% CI)	AUROCC^a (95% CI)	AUPRC^b (95% CI)
Physical
	BERT^c	0.593 (0.468-0.717)	0.427 (0.315-0.538)	0.929 (0.901-0.956)	0.832 (0.794-0.867)	0.496 (0.384-0.593)	0.775 (0.723-0.848)	0.537 (0.443-0.634)
	Word2vec/SVM^d	0.600 (0.286-0.900)	0.073 (0.026-0.136)	0.988 (0.974-0.997)	0.810 (0.770-0.848)	0.130 (0.048-0.227)	0.726 (0.670-0.780)	0.375 (0.224-0.474)
	Word2vec/XGBoost^e	0.595 (0.432-0.773)	0.268 (0.169-0.364)	0.956 (0.934-0.977)	0.822 (0.784-0.858)	0.370 (0.250-0.474)	0.726 (0.665-0.798)	0.461 (0.338-0.575)
Cognitive
	BERT	0.803 (0.696-0.895)	0.757 (0.652-0.854)	0.963 (0.941-0.981)	0.929 (0.903-0.953)	0.779 (0.697-0.855)	0.948 (0.922-0.979)	0.855 (0.791-0.930)
	Word2vec/SVM	0.829 (0.690-0.946)	0.414 (0.292-0.535)	0.983 (0.968-0.994)	0.889 (0.861-0.917)	0.552 (0.418-0.657)	0.917 (0.886-0.951)	0.730 (0.632-0.855)
	Word2vec/XGBoost	0.767 (0.625-0.884)	0.471 (0.359-0.586)	0.972 (0.953-0.988)	0.889 (0.858-0.917)	0.584 (0.468-0.684)	0.860 (0.817-0.924)	0.659 (0.550-0.782)
Social
	BERT	0.679 (0.500-0.848)	0.422 (0.289-0.568)	0.976 (0.960-0.990)	0.917 (0.891-0.943)	0.521 (0.379-0.658)	0.796 (0.704-0.912)	0.561 (0.434-0.741)
	Word2vec/SVM	0.778 (0.429-1.000)	0.156 (0.057-0.267)	0.995 (0.987-1.000)	0.905 (0.877-0.929)	0.259 (0.102-0.406)	0.817 (0.756-0.881)	0.393 (0.203-0.534)
	Word2vec/XGBoost	0.571 (0.286-0.833)	0.178 (0.068-0.300)	0.984 (0.971-0.995)	0.898 (0.868-0.924)	0.271 (0.118-0.415)	0.780 (0.706-0.850)	0.330 (0.154-0.436)

^aAUROCC: area under the receiver operating characteristic curve.

^bAUPRC: area under precision-recall curve.

^cBERT: bidirectional encoder representations from transformers.

^dSVM: support vector machine.

^eXGBoost: extreme gradient boosting.