Table 10. Comparison of fine-tuned BERT with benchmarks for multi-class classification (precision and recall).
| Disasters | Model | Precision | Recall | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Infra | Human | Non | Macro | Infra | Human | Non | Macro | ||
| Iraq–Iran Earthquake | Rudra et al. (2018) | 0 | 74.53 | 56.99 | 43.84 | 0 | 66.92 | 77.73 | 48.22 |
| Word2vec+RF | 57.14 | 76.32 | 80.00 | 71.15 | 38.10 | 93.55 | 47.06 | 59.57 | |
| TF-IDF+SVM | 99.0 | 88.89 | 86.25 | 91.71 | 16.67 | 66.67 | 99.0 | 60.78 | |
| TF-IDF+LSTM | 79.12 | 78.69 | 87.64 | 81.81 | 40.89 | 84.79 | 92.55 | 72.74 | |
| TF-IDF+Bi-LSTM | 82.38 | 81.21 | 88.24 | 83.94 | 43.19 | 85.58 | 92.60 | 73.79 | |
| TF-IDF+CNN | 83.15 | 82.76 | 89.74 | 85.21 | 45.10 | 86.83 | 92.68 | 74.86 | |
| Madichetty & Sridevi (2021) | 88.33 | 79.88 | 74.50 | 80.90 | 66.67 | 80.66 | 77.27 | 74.87 | |
| Proposed | 66.67 | 95.00 | 94.55 | 85.40 | 50.00 | 97.44 | 94.55 | 80.66 | |
| Sri Lanka Floods | Rudra et al. (2018) | 42.08 | 63.33 | 64.94 | 56.78 | 44.50 | 70.50 | 37.50 | 50.83 |
| Word2vec+RF | 57.14 | 76.32 | 80.00 | 71.15 | 38.10 | 93.55 | 47.06 | 59.57 | |
| TF-IDF+RF | 50.00 | 87.50 | 81.93 | 73.14 | 25.00 | 56.00 | 97.14 | 59.38 | |
| TF-IDF+LSTM | 58.45 | 87.90 | 82.69 | 76.34 | 41.49 | 84.10 | 88.34 | 71.31 | |
| TF-IDF+Bi-LSTM | 59.78 | 88.24 | 83.87 | 77.29 | 43.58 | 84.58 | 89.46 | 72.54 | |
| TF-IDF+CNN | 60.42 | 88.69 | 84.79 | 77.96 | 45.69 | 85.69 | 90.67 | 74.01 | |
| Madichetty & Sridevi (2021) | 85.33 | 84.83 | 78.02 | 82.73 | 70.50 | 70.50 | 93.00 | 78.0 | |
| Proposed | 61.11 | 89.47 | 99.20 | 83.26 | 100.00 | 94.44 | 93.23 | 95.89 | |
| Mexico Earthquake | Rudra et al. (2018) | 50.59 | 69.00 | 50.32 | 39.77 | 0 | 59.61 | 85.98 | 48.53 |
| TF-IDF+SVM | 87.50 | 90.91 | 91.63 | 90.01 | 38.89 | 68.97 | 99.49 | 69.12 | |
| Word2vec+RF | 57.14 | 76.32 | 80.00 | 71.15 | 38.10 | 93.55 | 47.06 | 59.57 | |
| TF-IDF+LSTM | 59.38 | 77.71 | 81.49 | 72.86 | 39.56 | 93.78 | 47.43 | 60.25 | |
| TF-IDF+Bi-LSTM | 59.85 | 78.32 | 82.38 | 73.51 | 40.32 | 93.97 | 49.39 | 61.22 | |
| TF-IDF+CNN | 60.12 | 79.78 | 83.58 | 74.49 | 41.58 | 94.12 | 51.78 | 62.49 | |
| Madichetty & Sridevi (2021) | 66.02 | 73.05 | 67.71 | 68.93 | 58.27 | 70.16 | 74.25 | 67.56 | |
| Proposed | 53.33 | 86.67 | 94.39 | 78.13 | 61.54 | 59.09 | 96.65 | 72.43 | |
| California Wildfires | Rudra et al. (2018) | 33.15 | 54.16 | 72.78 | 53.36 | 51.00 | 50.32 | 43.76 | 48.36 |
| TF-IDF+SVM | 71.43 | 100.0 | 84.59 | 85.34 | 16.13 | 57.89 | 99.56 | 57.86 | |
| Word2vec+RF | 57.14 | 76.32 | 80.00 | 71.15 | 38.10 | 93.55 | 47.06 | 59.57 | |
| TF-IDF+LSTM | 72.59 | 99.10 | 85.19 | 85.62 | 51.29 | 70.45 | 82.87 | 68.20 | |
| TF-IDF+Bi-LSTM | 73.39 | 99.31 | 86.49 | 86.39 | 51.78 | 73.59 | 83.29 | 69.55 | |
| TF-IDF+CNN | 74.19 | 99.43 | 87.69 | 87.10 | 52.10 | 75.98 | 85.48 | 71.18 | |
| Madichetty & Sridevi (2021) | 59.98 | 72.14 | 57.67 | 63.26 | 51.69 | 66.87 | 66.95 | 61.84 | |
| Proposed | 54.55 | 75.41 | 90.50 | 73.49 | 52.94 | 90.20 | 86.60 | 76.58 | |
| Hurricane Harvey | Rudra et al. (2018) | 42.42 | 46.22 | 49.98 | 46.21 | 73.23 | 7.92 | 40.06 | 40.40 |
| TF-IDF+SVM | 65.52 | 85.71 | 88.76 | 80.00 | 28.36 | 13.64 | 98.53 | 46.84 | |
| Word2vec+RF | 57.14 | 76.32 | 80.00 | 71.15 | 38.10 | 93.55 | 47.06 | 59.57 | |
| TF-IDF+LSTM | 66.59 | 76.57 | 81.23 | 74.79 | 39.49 | 93.71 | 48.39 | 60.53 | |
| TF-IDF+Bi-LSTM | 66.74 | 76.75 | 81.46 | 74.98 | 40.36 | 93.85 | 50.25 | 61.48 | |
| TF-IDF+CNN | 67.21 | 77.04 | 81.76 | 75.33 | 41.59 | 94.10 | 51.59 | 62.42 | |
| Madichetty & Sridevi (2021) | 73.34 | 64.31 | 67.17 | 68.27 | 64.97 | 55.25 | 79.55 | 66.59 | |
| Proposed | 53.98 | 63.49 | 94.32 | 70.60 | 71.76 | 71.43 | 89.25 | 77.48 | |
| Hurricane Maria | Rudra et al. (2018) | 45.15 | 0 | 53.63 | 32.97 | 76.90 | 0 | 40.34 | 39.08 |
| Word2vec+RF | 60.00 | 76.62 | 72.73 | 69.78 | 40.91 | 93.65 | 44.44 | 59.67 | |
| TF-IDF+SVM | 77.78 | 100.0 | 91.83 | 89.87 | 14.00 | 12.50 | 99.72 | 42.07 | |
| TF-IDF+LSTM | 78.10 | 98.21 | 91.95 | 89.42 | 18.45 | 20.38 | 90.10 | 42.97 | |
| TF-IDF+Bi-LSTM | 78.48 | 98.43 | 92.13 | 89.68 | 20.27 | 22.87 | 90.56 | 44.56 | |
| TF-IDF+CNN | 78.69 | 98.65 | 92.37 | 89.90 | 32.78 | 25.76 | 92.18 | 50.24 | |
| Madichetty & Sridevi (2021) | 77.16 | 71.17 | 74.73 | 74.35 | 75.52 | 48.63 | 85.52 | 69.89 | |
| Proposed | 47.37 | 42.86 | 97.69 | 62.64 | 54.55 | 69.23 | 95.99 | 73.26 | |
| Hurricane Irma | Rudra et al. (2018) | 50.17 | 0 | 57.10 | 35.76 | 73.95 | 0 | 52.47 | 42.14 |
| TF-IDF+SVM | 61.11 | 57.14 | 88.31 | 68.86 | 14.86 | 11.43 | 99.13 | 41.81 | |
| Word2vec+RF | 56.25 | 76.32 | 61.54 | 64.70 | 40.91 | 89.23 | 44.44 | 58.19 | |
| TF-IDF+LSTM | 61.49 | 77.38 | 88.53 | 75.80 | 42.41 | 88.49 | 44.78 | 58.56 | |
| TF-IDF+Bi-LSTM | 62.21 | 77.79 | 89.11 | 76.37 | 43.89 | 89.34 | 45.29 | 59.50 | |
| TF-IDF+CNN | 62.54 | 78.10 | 89.34 | 76.66 | 44.11 | 89.67 | 45.54 | 59.77 | |
| Madichetty & Sridevi (2021) | 75.24 | 59.81 | 71.79 | 68.95 | 76.19 | 43.42 | 77.92 | 65.84 | |
| Proposed | 48.39 | 20.00 | 96.48 | 54.96 | 68.18 | 17.65 | 94.36 | 60.06 | |
Notes.
The bold values are the highest performances achieved by the proposed model for each disaster.