Algorithm 1: Algorithm flow of the proposed hierarchical deep fusion framework.

Input: activity sets $\mathcal{A}=\{A_1,A_2,\dots ,A_n\}$; motion state sets $ℳ=\{M_1,M_2,\dots ,M_m\}$; correspondence ${\mathcal{C}}_{\mathcal{A}\leftrightarrow ℳ}$;  IMU sequence ${\mathit{x}}_t$; single frame $I_t$ or image sequence ${\mathit{I}}_t$ (corresponding to ${\mathit{x}}_t$);  pre-trained $LST{M}^{\mathit{x}}$ (for ${\mathit{x}}_t$); pre-trained $VG{G}_{conv}$; pre-trained $F{C}_j^I,j=1,2,\dots ,m$ (for $I_t$);  pre-trained $LST{M}_j^{\mathit{I}},j=1,2,\dots ,m$ (for ${\mathit{I}}_t$); pre-trained $F{C}_j^{LSTM},j=1,2,\dots ,m$ (for $LST{M}_j^{\mathit{I}}$);  indicator of the photo stream Ips (H or L) Output: activity index $k,k\in \{1,2,\dots ,n\}$

(1)Input ${\mathit{x}}_t$ into $LST{M}^{\mathit{x}}$ (2)Get the grouping index j of ${\mathit{x}}_t$ with Equation (6) (3)If Ips==L // Input photo stream is a low-frame-rate photo stream (${\mathit{x}}_t$ corresponds to single frame $I_t$) (4) Input $I_t$ into $VG{G}_{conv}$ with $F{C}_j^I$        //CNN as shown in Figure 6 (5) Get k with Equation (7) (6)Else   // Input photo stream is a high-frame-rate photo stream (${\mathit{x}}_t$ corresponds to image sequence ${\mathit{I}}_t$) (7) Input ${\mathit{I}}_t$ into $VG{G}_{conv}+LST{M}_j^{\mathit{I}}$ with $F{C}_j^{LSTM}$  //CNN+LSTM as shown in Figure 7 (8) Get k with Equation (8) (9)end If