Algorithm 2 Our Sensor Fusion Emotion Recognition (SFER) Algorithm.

Require: Variable for driver distinguish(Driver ID): $ID$

List of defined emotional state categories for driver: C

The recognized valence and arousal level of driver by facial expressions at time t: $({\widehat{\mathbf{V}}}_t,{\widehat{\mathbf{A}}}_t)$

The measured EDA response of driver at time t: $E_t$

Ensure: The recognized real emotion of driver at time t: ${\widehat{Y}}_t$

Initialize:

Let the sensor fusion emotion recognition model be SFER_Model

Let SFER_Model be the DNN

Let final hidden layer of SFER_Model be fully connected layer which n units and softmax function as activation function

${\widehat{\mathbf{V}}}^{ID}=\left[\right],{\widehat{\mathbf{A}}}^{ID}=\left[\right],E^{ID}=\left[\right]$

if Accumulated data for the driver with $ID$ exists then

Load ${\widehat{\mathbf{V}}}^{ID},{\widehat{\mathbf{A}}}^{ID},E^{ID}$

end if

while Driving exists do

# Accumulate data per each individual for personalization

${\widehat{\mathbf{V}}}^{ID}.insert\left({\widehat{\mathbf{V}}}_t\right)$,   ${\widehat{\mathbf{A}}}^{ID}.insert\left({\widehat{\mathbf{A}}}_t\right)$,  $E^{ID}.insert\left(E_t\right)$

# Mean subtraction

$({\widehat{\mathbf{V}}}_t,{\widehat{\mathbf{A}}}_t,E_t)-=(mean\left({\widehat{\mathbf{V}}}^{ID}\right),mean\left({\widehat{\mathbf{A}}}^{ID}\right),mean\left(E^{ID}\right))$

# Normalization

${\mathbf{X}}_t^T=(\frac{{\widehat{\mathbf{V}}}_t-min\left({\widehat{\mathbf{V}}}^{ID}\right)}{max\left({\widehat{\mathbf{V}}}^{ID}\right)-min\left({\widehat{\mathbf{V}}}^{ID}\right)},\frac{{\widehat{\mathbf{A}}}_t-min\left({\widehat{\mathbf{A}}}^{ID}\right)}{max\left({\widehat{\mathbf{A}}}^{ID}\right)-min\left({\widehat{\mathbf{A}}}^{ID}\right)},\frac{E_t-min\left(E^{ID}\right)}{max\left(E^{ID}\right)-min\left(E^{ID}\right)})$

$\widehat{{\mathbf{Y}}_{\mathbf{t}}^{𝕝}}=$SFER_Model(${\mathbf{X}}_{\mathbf{t}}$)

${\widehat{Y}}_t=C\left[argmax\left(\widehat{{\mathbf{Y}}_{\mathbf{t}}^{𝕝}}\right)\right]$

end while