Algorithm 1 Multiscale Appearance Dictionary Learning

Require: appearance vector samples ${\{{\mathbf{Y}}_{t-1}^1\}}_k={\{{\mathbf{y}}_{1,i}^k\}}_{i=1}^{M_1}$ and ${\{{\mathbf{Y}}_{t-1}^2\}}_k={\{{\mathbf{y}}_{2,j}^k\}}_{j=1}^{M_2}$, k = 1,…, S, initial dictionaries ${\{{\mathbf{D}}_{t-1}^1,{\mathbf{D}}_{t-1}^2\}}_z,z=1,\dots ,J$, and sparsity factor T.

${\mathbf{w}}_1^1={\{w_{1,i}^1\}}_{i=1}^{M_1}=\mathbf{1}$, ${\mathbf{w}}_2^1={\{w_{2,j}^1\}}_{j=1}^{M_2}=\mathbf{1}$.

for z = 1,…,J do k = S if z%S = 0; k = z%S, otherwise.

Resampling: Draw sample sets ${\stackrel{\sim }{\mathbf{Y}}}_1^z$ from ${\{{\mathbf{Y}}_{t-1}^1\}}_k$ and ${\stackrel{\sim }{\mathbf{Y}}}_2^z$ from ${\{{\mathbf{Y}}_{t-1}^2\}}_k$ based on distributions ${\mathbf{p}}_1^z={\{p_{1,i}^z\}}_{i=1}^{M_1}=\frac{{\mathbf{w}}_1^z}{{\sum }_{i=1}^{M_1}{w}_{1,i}^z}$ and ${\mathbf{p}}_2^z={\{p_{2,j}^z\}}_{j=1}^{M_2}=\frac{{\mathbf{w}}_2^z}{{\sum }_{j=2}^{M_2}{w}_{2,j}^z}$.

Dictionary Update: Apply the K-SVD to learn ${\{{\mathbf{D}}_t^1,{\mathbf{D}}_t^2\}}_z$ from ${\stackrel{\sim }{\mathbf{Y}}}_1^z$ and ${\stackrel{\sim }{\mathbf{Y}}}_2^z$: $\underset{{\{{\mathbf{D}}_t^c\}}_{z,}\mathbf{X}}{min}{\Vert {\stackrel{\sim }{\mathbf{Y}}}_c^z-{\{{\mathbf{D}}_t^c\}}_z\mathbf{X}\Vert }_2^2\mathrm{s}.\mathrm{t}.{\forall }_i,{\Vert {\mathbf{x}}_i\Vert }_0\le T;c\in \{1,2\}.$

Sparse Coding: $\forall \mathbf{y}\in {\{{\mathbf{Y}}_{t-1}^1,{\mathbf{Y}}_{t-1}^2\}}_k$, solve for the sparse representations with respect to ${\{{\mathbf{D}}_t^1\}}_z$ and ${\{{\mathbf{D}}_t^2\}}_z$ using the OMP, and get residues $R{(\mathbf{y},{\mathbf{D}}_t^1)}_z$ and $R{(\mathbf{y},{\mathbf{D}}_t^2)}_z$.

Classification: Make a hypothesis $h_z:\mathbf{y}\in {\{{\mathbf{Y}}_{t-1}^1,{\mathbf{Y}}_{t-1}^2\}}_k\to \{0,1\}:h_z(\mathbf{y})=\mathit{\text{Heaviside}}(R{(\mathbf{y},{\mathbf{D}}_t^2)}_z-R{(\mathbf{y},{\mathbf{D}}_t^1)}_z)$. Calculate the error of $h_z:{\in }_z=\sum _{i=1}^{M_1}{p}_{1,i}^z|h_z({\mathbf{y}}_{1,i}^k)-1|+\sum _{j=1}^{M_2}{p}_{2,j}^z{h}_z({\mathbf{y}}_{2,j}^k)$. Set βz = ∈z/(1 − ∈z).

Weight Update: $w_{1,i}^{z+1}=w_{1,i}^z{\beta }_z^{1-|h_z({\mathbf{y}}_{1,i}^k)-1|},w_{2,j}^{z+1}=w_{2,j}^z{\beta }_z^{1-h_z({\mathbf{y}}_{2,j}^k)}$

end for dictionary pairs ${\{{\mathbf{D}}_t^1,{\mathbf{D}}_t^2\}}_z$, weighting parameters βz, z = 1,…,J.