Table 4.
Comparison of different cross-validation methods and different performance metrics (column) for different models on two sensor combinations (row).
| Sensors | Model | k-fold | leave-recordings-out | leave-one-subject-out | |||
|---|---|---|---|---|---|---|---|
| Acc | F1 | Acc | F1 | Acc | F1 | ||
| {LW, RW} | CNN-LSTM | 0.4 (±0.01) | 0.36 (±0.02) | 0.41 (±0.04) | 0.37 (±0.04) | 0.4 (±0.05) | 0.35 (±0.06) |
| {LW, RW} | ResNet | 0.34 (±0.06) | 0.33 (±0.07) | 0.35 (±0.02) | 0.33 (±0.03) | 0.35 (±0.07) | 0.33 (±0.06) |
| {LW, RW} | DeepConvLSTM | 0.32 (±0.03) | 0.21 (±0.04) | 0.27 (±0.06) | 0.17 (±0.05) | 0.31 (±0.03) | 0.19 (±0.02) |
| {LW, ST} | CNN-LSTM | 0.41 (±0.05) | 0.38 (±0.06) | 0.42 (±0.06) | 0.38 (±0.06) | 0.4 (±0.03) | 0.34 (±0.03) |
| {LW, ST} | ResNet | 0.34 (±0.05) | 0.32 (±0.03) | 0.37 (±0.05) | 0.34 (±0.05) | 0.35 (±0.06) | 0.33 (±0.05) |
| {LW, ST} | DeepConvLSTM | 0.32 (±0.07) | 0.24 (±0.09) | 0.32 (±0.05) | 0.25 (±0.06) | 0.33 (±0.03) | 0.25 (±0.04) |
| {LW, LF} | CNN-LSTM | 0.38 (±0.03) | 0.34 (±0.03) | 0.41 (±0.05) | 0.36 (±0.04) | 0.4 (±0.02) | 0.36 (±0.03) |
| {LW, LF} | ResNet | 0.31 (±0.04) | 0.3 (±0.04) | 0.33 (±0.04) | 0.31 (±0.04) | 0.34 (±0.04) | 0.32 (±0.03) |
| {LW, LF} | DeepConvLSTM | 0.32 (±0.03) | 0.23 (±0.03) | 0.29 (±0.04) | 0.2 (±0.06) | 0.32 (±0.06) | 0.23 (±0.06) |
| {LW, RF} | CNN-LSTM | 0.42 (±0.04) | 0.37 (±0.04) | 0.42 (±0.04) | 0.37 (±0.04) | 0.43 (±0.04) | 0.38 (±0.04) |
| {LW, RF} | ResNet | 0.36 (±0.03) | 0.34 (±0.04) | 0.33 (±0.04) | 0.32 (±0.04) | 0.34 (±0.04) | 0.33 (±0.05) |
| {LW, RF} | DeepConvLSTM | 0.31 (±0.04) | 0.23 (±0.03) | 0.3 (±0.06) | 0.22 (±0.05) | 0.32 (±0.04) | 0.24 (±0.06) |
| {RW, ST} | CNN-LSTM | 0.38 (±0.04) | 0.32 (±0.04) | 0.39 (±0.06) | 0.32 (±0.05) | 0.38 (±0.03) | 0.32 (±0.04) |
| {RW, ST} | ResNet | 0.36 (±0.08) | 0.33 (±0.07) | 0.32 (±0.03) | 0.32 (±0.04) | 0.32 (±0.06) | 0.31 (±0.06) |
| {RW, ST} | DeepConvLSTM | 0.31 (±0.05) | 0.22 (±0.06) | 0.3 (±0.04) | 0.22 (±0.04) | 0.3 (±0.02) | 0.21 (±0.04) |
| {RW, LF} | CNN-LSTM | 0.39 (±0.07) | 0.34 (±0.07) | 0.38 (±0.04) | 0.31 (±0.03) | 0.39 (±0.06) | 0.33 (±0.05) |
| {RW, LF} | ResNet | 0.32 (±0.03) | 0.31 (±0.03) | 0.31 (±0.04) | 0.3 (±0.04) | 0.31 (±0.06) | 0.3 (±0.06) |
| {RW, LF} | DeepConvLSTM | 0.32 (±0.05) | 0.22 (±0.04) | 0.31 (±0.05) | 0.21 (±0.05) | 0.3 (±0.03) | 0.21 (±0.03) |
| {RW, RF} | CNN-LSTM | 0.39 (±0.06) | 0.33 (±0.06) | 0.39 (±0.02) | 0.33 (±0.03) | 0.38 (±0.05) | 0.32 (±0.05) |
| {RW, RF} | ResNet | 0.34 (±0.07) | 0.32 (±0.08) | 0.32 (±0.06) | 0.31 (±0.07) | 0.34 (±0.05) | 0.32 (±0.05) |
| {RW, RF} | DeepConvLSTM | 0.3 (±0.04) | 0.21 (±0.04) | 0.3 (±0.02) | 0.21 (±0.02) | 0.31 (±0.07) | 0.21 (±0.07) |
| {ST, LF} | CNN-LSTM | 0.33 (±0.05) | 0.28 (±0.05) | 0.32 (±0.03) | 0.26 (±0.02) | 0.32 (±0.05) | 0.27 (±0.05) |
| {ST, LF} | ResNet | 0.27 (±0.03) | 0.26 (±0.03) | 0.24 (±0.05) | 0.23 (±0.05) | 0.22 (±0.05) | 0.21 (±0.05) |
| {ST, LF} | DeepConvLSTM | 0.31 (±0.06) | 0.23 (±0.06) | 0.29 (±0.03) | 0.21 (±0.03) | 0.28 (±0.03) | 0.19 (±0.03) |
| {ST, RF} | CNN-LSTM | 0.34 (±0.07) | 0.28 (±0.08) | 0.34 (±0.08) | 0.28 (±0.09) | 0.34 (±0.01) | 0.27 (±0.02) |
| {ST, RF} | ResNet | 0.25 (±0.04) | 0.23 (±0.04) | 0.28 (±0.05) | 0.26 (±0.06) | 0.28 (±0.05) | 0.26 (±0.05) |
| {ST, RF} | DeepConvLSTM | 0.29 (±0.03) | 0.21 (±0.04) | 0.29 (±0.03) | 0.21 (±0.05) | 0.29 (±0.03) | 0.19 (±0.03) |
| {LF, RF} | CNN-LSTM | 0.33 (±0.04) | 0.27 (±0.05) | 0.33 (±0.04) | 0.28 (±0.04) | 0.31 (±0.03) | 0.26 (±0.02) |
| {LF, RF} | ResNet | 0.22 (±0.06) | 0.21 (±0.07) | 0.23 (±0.05) | 0.22 (±0.05) | 0.22 (±0.02) | 0.21 (±0.04) |
| {LF, RF} | DeepConvLSTM | 0.28 (±0.04) | 0.19 (±0.03) | 0.27 (±0.04) | 0.19 (±0.05) | 0.26 (±0.03) | 0.18 (±0.03) |
The best performing model for each sensor and validation setup is highlighted in bold.