Algorithm 1 CorrNet

Input: Training set with n instances in modality 1: $X_1={\left\{x_1^i\right\}}_{i=1}^n,x_1^i\in R^{L\times C_1}$ and modality 2: $X_2={\left\{x_2^i\right\}}_{i=1}^n,x_2^i\in R^{L\times C_2}$ Output: Fine-grained emotion labels (i.e., valence:$V_a={\left\{v_i\right\}}_{i=1}^n$ and arousal $A_r={\left\{a_i\right\}}_{i=1}^n$) 1:for j = 1 and 2 do 2:  Encoder→${\varphi }_j=X_j\otimes \psi (\omega ,c)$ 3:  Decoder→${\eta }_j=\varphi \overline{\otimes }{\psi }^{\prime }(C_j,c)$ 4:end for 5:Group instances according to video stimulus: 6:for t in T = number of video stimulus do 7:  $(H_1^t,H_2^t)=\mathbf{CCA}({\psi }_1^t,{\psi }_2^t)$ 8:  $F^t=[{\psi }_1^t·H_1^t,{\psi }_2^t·H_2^t]$ 9:end for 10:$F={\left\{F^t\right\}}_{t=1}^T,F\in R^{n\times k}$ 11:${(a_i,v_i)}_{i=1}^n=\mathbf{BLS}\left(F\right)$