Table 3.
Comprehensive 10-fold cross-validated mean performance evaluation metrics with standard deviations for various models and embeddings, including time performance (training and inference times).
| Word embedding | Model | Macro average precision | Macro average recall | Macro average F1-score | Accuracy | Training time (s) | Inference time (s) |
|---|---|---|---|---|---|---|---|
| BoW | RF | 75% ± 1% | 61% ± 2% | 61% ± 2% | 67% ± 1% | 6.73 | 0.03 |
| BoW | NB | 63% ± 1% | 62% ± 2% | 62% ± 2% | 64% ± 1% | 0.82 | 0.01 |
| BoW | SVM | 76% ± 2% | 76% ± 2% | 76% ± 2% | 77% ± 1% | 4.35 | 0.30 |
| BoW | GBM | 82% ± 1% | 79% ± 1% | 79% ± 1% | 81% ± 1% | 262.81 | 0.02 |
| BoW | LGBM | 83% ± 1% | 81% ± 1% | 81% ± 1% | 83% ± 1% | 35.54 | 0.01 |
| BoW | XGBoost | 77% ± 0% | 75% ± 1% | 75% ± 1% | 78% ± 0% | 86.50 | 0.02 |
| BoW | Catboost | 80% ± 0% | 74% ± 1% | 75% ± 1% | 77% ± 1% | 536.16 | 0.02 |
| BoW | LGBM+KNN+MLP | 82% ± 1% | 81% ± 2% | 81% ± 2% | 84% ± 0% | 1422.53 | 45.43 |
| BoW | RF+KNN+MLP | 74% ± 1% | 73% ± 2% | 74% ± 1% | 75% ± 1% | 1212.43 | 40.54 |
| BoW | GBM+RF Stacking Classifier | 78% ± 0% | 76% ± 1% | 76% ± 1% | 78% ± 0% | 1364.06 | 50.57 |
| BoW | GBM+RF Voting Classifier | 78% ± 0% | 69% ± 1% | 70% ± 1% | 72% ± 1% | 1250.37 | 48.30 |
| BoW | Single Hidden Layer NN | 71% ± 4% | 69% ± 4% | 69% ± 4% | 72% ± 4% | 646.34 | 24.53 |
| BoW | 3 Hidden Layers NN | 68% ± 4% | 67% ± 4% | 67% ± 4% | 70% ± 4% | 904.64 | 25.35 |
| TFIDF | RF | 69% ± 1% | 60% ± 3% | 60% ± 3% | 66% ± 2% | 6.75 | 0.04 |
| TFIDF | NB | 65% ± 1% | 56% ± 3% | 56% ± 3% | 61% ± 2% | 0.89 | 0.02 |
| TFIDF | SVM | 74% ± 2% | 70% ± 3% | 70% ± 3% | 73% ± 2% | 5.15 | 0.46 |
| TFIDF | GBM | 80% ± 1% | 77% ± 3% | 77% ± 3% | 79% ± 1% | 273.51 | 0.05 |
| TFIDF | LGBM | 80% ± 0% | 77% ± 3% | 78% ± 3% | 80% ± 1% | 36.12 | 0.02 |
| TFIDF | XGBoost | 68% ± 1% | 67% ± 1% | 67% ± 1% | 70% ± 1% | 87.22 | 0.03 |
| TFIDF | Catboost | 74% ± 1% | 72% ± 1% | 72% ± 1% | 74% ± 1% | 542.44 | 0.03 |
| TFIDF | LGBM+KNN+MLP | 79% ± 0% | 77% ± 1% | 77% ± 1% | 79% ± 0% | 1532.37 | 46.24 |
| TFIDF | RF Bagging | 76% ± 0% | 48% ± 1% | 43% ± 1% | 56% ± 1% | 764.34 | 35.34 |
| TFIDF | RF+KNN+MLP | 75% ± 1% | 71% ± 3% | 72% ± 2% | 74% ± 0% | 1254.65 | 42.75 |
| TFIDF | GBM+RF Stacking Classifier | 77% ± 0% | 76% ± 1% | 76% ± 1% | 78% ± 0% | 1352.53 | 52.65 |
| TFIDF | GBM+RF Voting Classifier | 75% ± 0% | 69% ± 2% | 70% ± 1% | 73% ± 0% | 1283.23 | 50.75 |
| TFIDF | Single Hidden Layer NN | 68% ± 3% | 67% ± 4% | 67% ± 4% | 69% ± 2% | 650.34 | 22.43 |
| TFIDF | 3 Hidden Layers NN | 93% ± 3% | 92% ± 4% | 92% ± 3% | 93% ± 4% | 954.64 | 27.53 |
| word2vec | RF | 51% ± 1% | 49% ± 2% | 49% ± 2% | 56% ± 1% | 6.82 | 0.05 |
| word2vec | NB | 41% ± 1% | 34% ± 1% | 19% ± 1% | 36% ± 1% | 1.03 | 0.04 |
| word2vec | SVM | 67% ± 0% | 49% ± 2% | 45% ± 3% | 57% ± 1% | 5.78 | 0.57 |
| word2vec | GBM | 51% ± 1% | 50% ± 2% | 50% ± 2% | 56% ± 1% | 282.10 | 0.08 |
| word2vec | LGBM | 53% ± 1% | 49% ± 1% | 47% ± 1% | 57% ± 1% | 37.41 | 0.03 |
| word2vec | XGBoost | 55% ± 1% | 50% ± 2% | 48% ± 2% | 56% ± 1% | 88.60 | 0.03 |
| word2vec | Catboost | 71% ± 0% | 49% ± 1% | 45% ± 1% | 57% ± 1% | 553.37 | 0.04 |
| word2vec | LGBM+KNN+MLP | 53% ± 1% | 46% ± 2% | 41% ± 3% | 53% ± 1% | 1448.76 | 47.34 |
| word2vec | RF+KNN+MLP | 55% ± 1% | 49% ± 3% | 43% ± 3% | 57% ± 1% | 1345.53 | 55.23 |
| word2vec | GBM+RF Stacking Classifier | 46% ± 2% | 45% ± 2% | 45% ± 2% | 51% ± 1% | 1412.64 | 53.73 |
| word2vec | GBM+RF Voting Classifier | 47% ± 1% | 47% ± 1% | 47% ± 1% | 53% ± 1% | 1350.23 | 28.78 |
| word2vec | Single Hidden Layer NN | 36% ± 5% | 45% ± 4% | 40% ± 4% | 53% ± 3% | 704.65 | 28.34 |
| word2vec | 3 Hidden Layers NN | 38% ± 4% | 48% ± 4% | 42% ± 4% | 56% ± 3% | 1034.89 | 33.38 |
| BERT | RF | 53% ± 2% | 51% ± 3% | 49% ± 3% | 58% ± 1% | 805.43 | 45.53 |
| BERT | NB | 50% ± 3% | 51% ± 2% | 50% ± 2% | 53% ± 1% | 1.12 | 0.07 |
| BERT | SVM | 52% ± 2% | 52% ± 2% | 52% ± 2% | 56% ± 1% | 6.12 | 0.76 |
| BERT | GBM | 56% ± 1% | 54% ± 2% | 54% ± 2% | 60% ± 2% | 291.11 | 0.11 |
| BERT | LGBM | 58% ± 2% | 57% ± 2% | 57% ± 2% | 61% ± 2% | 37.66 | 0.03 |
| BERT | XGBoost | 56% ± 2% | 56% ± 2% | 56% ± 2% | 60% ± 2% | 89.10 | 0.04 |
| BERT | Catboost | 56% ± 2% | 49% ± 2% | 46% ± 2% | 57% ± 2% | 562.29 | 0.05 |
| BERT | LGBM+KNN+MLP | 61% ± 1% | 59% ± 2% | 59% ± 2% | 62% ± 1% | 1623.65 | 65.34 |
| BERT | RF+KNN+MLP | 60% ± 1% | 58% ± 2% | 57% ± 2% | 62% ± 1% | 1443.22 | 60.44 |
| BERT | GBM+RF Stacking Classifier | 55% ± 2% | 54% ± 2% | 54% ± 2% | 56% ± 1% | 1523.75 | 70.23 |
| BERT | GBM+RF Voting Classifier | 60% ± 1% | 58% ± 2% | 58% ± 2% | 66% ± 0% | 1452.45 | 67.85 |
| BERT | Single Hidden Layer NN | 61% ± 3% | 58% ± 4% | 59% ± 4% | 62% ± 2% | 945.53 | 35.64 |
| BERT | 3 Hidden Layers NN | 62% ± 4% | 60% ± 4% | 60% ± 4% | 64% ± 4% | 1305.39 | 45.49 |
| SBERT | RF | 61% ± 0% | 48% ± 3% | 44% ± 4% | 54% ± 2% | 6.95 | 0.06 |
| SBERT | NB | 53% ± 2% | 40% ± 3% | 32% ± 4% | 44% ± 2% | 1.14 | 0.09 |
| SBERT | SVM | 56% ± 1% | 57% ± 2% | 56% ± 2% | 58% ± 1% | 6.28 | 0.81 |
| SBERT | GBM | 55% ± 2% | 51% ± 2% | 49% ± 2% | 55% ± 2% | 302.43 | 0.15 |
| SBERT | LGBM | 55% ± 2% | 52% ± 2% | 50% ± 2% | 56% ± 2% | 38.18 | 0.04 |
| SBERT | XGBoost | 56% ± 2% | 56% ± 2% | 56% ± 2% | 57% ± 2% | 90.53 | 0.06 |
| SBERT | Catboost | 69% ± 0% | 48% ± 3% | 42% ± 4% | 53% ± 2% | 577.54 | 0.05 |
| SBERT | LGBM+KNN+MLP | 55% ± 2% | 49% ± 2% | 47% ± 2% | 53% ± 2% | 1734.23 | 68.64 |
| SBERT | RF+KNN+MLP | 56% ± 1% | 54% ± 2% | 54% ± 2% | 57% ± 1% | 1522.42 | 63.43 |
| SBERT | GBM+RF Stacking Classifier | 44% ± 2% | 44% ± 2% | 43% ± 2% | 48% ± 1% | 1623.86 | 72.57 |
| SBERT | GBM+RF Voting Classifier | 53% ± 2% | 50% ± 2% | 50% ± 2% | 54% ± 2% | 1553.54 | 74.67 |
| SBERT | Single Hidden Layer NN | 55% ± 3% | 55% ± 4% | 54% ± 3% | 59% ± 3% | 954.64 | 38.48 |
| SBERT | 3 Hidden Layers NN | 56% ± 4% | 57% ± 3% | 57% ± 3% | 59% ± 2% | 1402.54 | 48.48 |
| RoBERTa | RF | 62% ± 2% | 62% ± 2% | 62% ± 2% | 64% ± 1% | 7.23 | 0.08 |
| RoBERTa | NB | 63% ± 1% | 55% ± 2% | 52% ± 3% | 54% ± 2% | 1.16 | 1.05 |
| RoBERTa | SVM | 65% ± 1% | 65% ± 1% | 65% ± 1% | 67% ± 0% | 6.76 | 0.88 |
| RoBERTa | GBM | 63% ± 2% | 62% ± 3% | 63% ± 2% | 65% ± 2% | 314.44 | 0.17 |
| RoBERTa | LGBM | 64% ± 2% | 63% ± 3% | 64% ± 2% | 66% ± 1% | 38.89 | 0.05 |
| RoBERTa | XGBoost | 65% ± 1% | 63% ± 2% | 63% ± 2% | 66% ± 1% | 91.14 | 0.07 |
| RoBERTa | Catboost | 63% ± 2% | 62% ± 3% | 61% ± 4% | 64% ± 2% | 583.28 | 0.07 |
| RoBERTa | LGBM+KNN+MLP | 66% ± 2% | 66% ± 2% | 65% ± 3% | 66% ± 2% | 1823.93 | 72.23 |
| RoBERTa | RF+KNN+MLP | 58% ± 3% | 55% ± 2% | 55% ± 2% | 60% ± 1% | 1654.54 | 67.76 |
| RoBERTa | GBM+RF Stacking Classifier | 61% ± 3% | 61% ± 3% | 61% ± 3% | 63% ± 2% | 1705.36 | 75.96 |
| RoBERTa | GBM+RF Voting Classifier | 60% ± 2% | 60% ± 2% | 59% ± 3% | 60% ± 2% | 1653.78 | 73.47 |
| RoBERTa | Single Hidden Layer NN | 84% ± 4% | 84% ± 4% | 84% ± 4% | 84% ± 4% | 1349.46 | 154.39 |
| RoBERTa | 3 Hidden Layers NN | 84% ± 3% | 83% ± 4% | 83% ± 4% | 84% ± 3% | 1898.05 | 153.64 |
| RoBERTa | BiLSTM+3 Hidden Layers NN | 84% ± 3% | 84% ± 3% | 84% ± 3% | 85% ± 2% | 3404.54 | 148.43 |
| RoBERTa | BilSTM+CNN | 83% ± 2% | 81% ± 4% | 82% ± 3% | 83% ± 2% | 5328.73 | 178.64 |
| RoBERTa | Proposed TRABSA model | 94% ± 1% | 93% ± 2% | 94% ± 1% | 94% ± 1% | 3675.21 | 147.14 |