Table 9. Comparison of fine-tuned BERT with benchmarks for multi-class classification (accuracy and f1-score).
| Disasters | Model | Accuracy | F1-score | |||
|---|---|---|---|---|---|---|
| Infrastructure | Human | Non | Macro | |||
| Iraq–Iran Earthquake | Rudra et al. (2018) | – | 0 | 69.36 | 65.13 | 44.83 |
| Word2vec+RF | 74.00 | 45.71 | 84.06 | 59.26 | 63.01 | |
| TF-IDF+SVM | 86.87 | 28.57 | 76.19 | 92.62 | 65.79 | |
| TF-IDF+LSTM | 87.79 | 48.89 | 84.55 | 92.65 | 75.36 | |
| TF-IDF+Bi-LSTM | 88.27 | 50.19 | 85.34 | 92.70 | 76.07 | |
| TF-IDF+CNN | 89.29 | 53.10 | 86.28 | 92.78 | 77.38 | |
| Madichetty & Sridevi (2021) | – | 74.33 | 80.00 | 75.18 | 76.50 | |
| Proposed | 93.88 | 57.14 | 96.20 | 94.55 | 82.63 | |
| Sri Lanka Floods | Rudra et al. (2018) | – | 41.72 | 64.56 | 39.52 | 48.60 |
| Word2vec+RF | 74.00 | 45.71 | 84.06 | 59.26 | 63.01 | |
| TF-IDF+RF | 81.55 | 33.33 | 68.29 | 88.89 | 63.50 | |
| TF-IDF+LSTM | 82.32 | 47.64 | 84.18 | 89.71 | 73.84 | |
| TF-IDF+Bi-LSTM | 83.53 | 49.71 | 84.47 | 90.32 | 74.83 | |
| TF-IDF+CNN | 84.10 | 51.43 | 84.90 | 91.54 | 75.95 | |
| Madichetty & Sridevi (2021) | – | 74.61 | 74.69 | 83.59 | 77.63 | |
| Proposed | 93.83 | 75.86 | 91.89 | 96.12 | 87.96 | |
| Mexico Earthquake | Rudra et al. (2018) | – | 0 | 63.33 | 63.31 | 42.21 |
| Word2vec+RF | 74.00 | 45.71 | 84.06 | 59.26 | 63.01 | |
| TF-IDF+SVM | 91.43 | 53.85 | 78.43 | 95.40 | 75.89 | |
| TF-IDF+LSTM | 91.20 | 54.18 | 79.10 | 86.39 | 73.22 | |
| TF-IDF+Bi-LSTM | 91.30 | 54.58 | 79.40 | 87.10 | 73.69 | |
| TF-IDF+CNN | 91.33 | 55.38 | 79.90 | 87.37 | 74.21 | |
| Madichetty & Sridevi (2021) | – | 61.15 | 71.21 | 70.65 | 67.67 | |
| Proposed | 91.39 | 57.14 | 70.27 | 95.51 | 74.31 | |
| California Wildfires | Rudra et al. (2018) | – | 39.81 | 51.35 | 53.18 | 48.11 |
| Word2vec+RF | 74.00 | 45.71 | 84.06 | 59.26 | 63.01 | |
| TF-IDF+SVM | 85.42 | 26.32 | 73.33 | 91.46 | 63.70 | |
| TF-IDF+LSTM | 83.01 | 45.82 | 83.27 | 88.10 | 72.39 | |
| TF-IDF+Bi-LSTM | 83.12 | 46.29 | 83.43 | 88.21 | 72.64 | |
| TF-IDF+CNN | 83.27 | 46.76 | 83.64 | 88.38 | 72.92 | |
| Madichetty & Sridevi (2021) | – | 55.02 | 68.98 | 61.53 | 61.84 | |
| Proposed | 83.33 | 53.73 | 82.14 | 88.51 | 74.79 | |
| Hurricane Harvey | Rudra et al. (2018) | – | 53.59 | 13.01 | 43.97 | 36.86 |
| Word2vec+RF | 74.00 | 45.71 | 84.06 | 59.26 | 63.01 | |
| TF-IDF+SVM | 87.88 | 39.58 | 23.53 | 93.39 | 52.17 | |
| TF-IDF+LSTM | 85.58 | 55.37 | 64.29 | 91.87 | 70.51 | |
| TF-IDF+Bi-LSTM | 85.67 | 56.48 | 64.59 | 92.12 | 71.06 | |
| TF-IDF+CNN | 85.87 | 57.84 | 64.7 | 92.43 | 71.65 | |
| Madichetty & Sridevi (2021) | – | 68.52 | 58.69 | 72.54 | 66.58 | |
| Proposed | 86.11 | 61.62 | 67.23 | 91.71 | 73.52 | |
| Hurricane Maria | Rudra et al. (2018) | – | 0 | 77.36 | 77.87 | 51.74 |
| TF-IDF+SVM | 91.70 | 23.73 | 22.22 | 95.61 | 47.19 | |
| Word2vec+RF | 73.79 | 48.65 | 84.29 | 55.17 | 62.70 | |
| TF-IDF+LSTM | 91.79 | 49.58 | 43.32 | 95.87 | 62.92 | |
| TF-IDF+Bi-LSTM | 91.87 | 49.86 | 44.13 | 96.09 | 63.36 | |
| TF-IDF+CNN | 92.10 | 50.38 | 46.67 | 96.35 | 64.46 | |
| Madichetty & Sridevi (2021) | – | 75.92 | 57.13 | 79.14 | 70.73 | |
| Proposed | 93.84 | 50.7 | 52.94 | 96.84 | 66.83 | |
| Hurricane Irma | Rudra et al. (2018) | – | 59.5 | 0 | 54.08 | 37.86 |
| TF-IDF+SVM | 87.42 | 23.91 | 19.05 | 93.41 | 45.46 | |
| Word2vec+RF | 71.43 | 47.37 | 82.27 | 51.61 | 60.42 | |
| TF-IDF+LSTM | 88.58 | 29.7 | 40.58 | 93.59 | 54.62 | |
| TF-IDF+Bi-LSTM | 88.87 | 30.49 | 40.78 | 93.7 | 54.99 | |
| TF-IDF+CNN | 89.29 | 31.89 | 41.28 | 93.8 | 55.65 | |
| Madichetty & Sridevi (2021) | – | 75.26 | 49.89 | 74.34 | 66.50 | |
| Proposed | 91.24 | 56.6 | 18.75 | 95.41 | 56.92 | |
Notes.
The bold values are the highest performances achieved by the proposed model for each disaster.