Skip to main content

View full-text article in PMC

. 2021 Nov 3;23(11):e26777. doi: 10.2196/26777

Table 2.

Performance of natural language processing/machine learning models for pain interference domain by 3 symptom attributes.

Attributes and models		Precision (95% CI)	Sensitivity (95% CI)	Specificity (95% CI)	Accuracy (95% CI)	F1 (95% CI)	AUROCC^a (95% CI)	AUPRC^b (95% CI)
Physical
	BERT^c	0.692 (0.555-0.811)	0.507 (0.387-0.618)	0.950 (0.924-0.972)	0.870 (0.836-0.903)	0.585 (0.467-0.683)	0.875 (0.824-0.948)	0.677 (0.568-0.770)
	Word2vec/SVM^d	0.722 (0.562-0.867)	0.366 (0.262-0.479)	0.969 (0.948-0.987)	0.859 (0.824-0.893)	0.486 (0.362-0.594)	0.868 (0.826-0.922)	0.623 (0.5090.743)
	Word2vec/XGBoost^e	0.697 (0.528-0.857)	0.324 (0.221-0.435)	0.969 (0.949-0.987)	0.852 (0.813-0.887)	0.442 (0.318-0.551)	0.830 (0.769-0.888)	0.553 (0.437-0.659)
Cognitive
	BERT	0.800 (0.657-0.935)	0.583 (0.432-0.735)	0.980 (0.964-0.994)	0.931 (0.905-0.957)	0.675 (0.543-0.779)	0.923 (0.879-0.997)	0.818 (0.735-0.917)
	Word2vec/SVM	0.760 (0.583-0.920)	0.396 (0.254-0.533)	0.983 (0.967-0.994)	0.910 (0.882-0.939)	0.521 (0.361-0.648)	0.900 (0.863-0.957)	0.609 (0.434-0.761)
	Word2vec/XGBoost	0.769 (0.500-1.000)	0.208 (0.104-0.333)	0.991 (0.980-1.000)	0.895 (0.867-0.926)	0.328 (0.178-0.474)	0.828 (0.748-0.905)	0.474 (0.321-0.630)
Social
	BERT	0.636 (0.461-0.800)	0.500 (0.349-0.652)	0.966 (0.946-0.983)	0.916 (0.887-0.941)	0.560 (0.410-0.690)	0.857 (0.786-0.918)	0.566 (0.402-0.750)
	Word2vec/SVM	0.286 (0-0.668)	0.048 (0-0.118)	0.986 (0.973-0.997)	0.885 (0.854-0.916)	0.082 (0.035-0.200)	0.804 (0.742-0.878)	0.309 (0.173-0.426)
	Word2vec/XGBoost	0.556 (0.222-0.875)	0.119 (0.029-0.229)	0.989 (0.977-0.997)	0.895 (0.864-0.923)	0.196 (0.072-0.343)	0.786 (0.728-0.850)	0.304 (0.148-0.420)

^aAUROCC: area under the receiver operating characteristic curve.

^bAUPRC: area under precision-recall curve.

^cBERT: bidirectional encoder representations from transformers.

^dSVM: support vector machine.

^eXGBoost: extreme gradient boosting.