Algorithm 1.

Multi-Source Multi-Target Dictionary Learning (MMDL)

Input: Samples and corresponding labels from different time points: {X1, X2, .....XT} and {Y1, Y2, .....YT}
Output: The model for different time points: {W1, …, WT}.
1:	Stage 1: Multi-Source Dictionary Learning
2:	for k = 1 to κ do
3:	For each image patch xt(i) from sample Xt, i ∈ {1, …, nt} and t ∈ {1, …, T}.
4:	Update ${\widehat{D}}_t^k:{\widehat{D}}_t^k=\mathit{\Phi }$.
5:	Update $z_t^{k+1}(i)$ and index set $I_t^{k+1}(i)$ by a few steps of CCD:
6:	$[z_t^{k+1}(i),I_t^{k+1}(i)]=\mathit{CCD}(D_t^k,{\overline{D}}_t^k,x_t(i),I_t^k(i),z_t^k(i))$.
7:	Update the D̂t and D̄t by one step SGD:
8:	$[{\widehat{D}}_t^{k+1},{\overline{D}}_t^{k+1}]=\mathit{SGD}({\widehat{D}}_t^k,{\overline{D}}_t^k,x_t(i),I_t^{k+1}(i),z_t^{k+1}(i))$.
9:	Normalize ${\widehat{D}}_t^{k+1}$ and ${\overline{D}}_t^{k+1}$ based on the index set $I_t^{k+1}(i)$.
10:	Update the shared dictionary $\mathit{\Phi }:\mathit{\Phi }={\widehat{D}}_t^{k+1}$.
11:	end for
12:	Obtain the learnt dictionaries and sparse codes: {D1, …, DT}, {Z1, …, ZT}.
13:	Stage 2: Multi-Target Regression with incomplete label
14:	for t = 1 to T do
15:	Given the jth column Yt(j) in Yt, for the jth model wt(j) in Wt
16:	$w_t(j)={({\stackrel{\sim }{Z}}_t{\stackrel{\sim }{Z}}_t^T+\xi I)}^{-1}{\stackrel{\sim }{Z}}_t{\stackrel{\sim }{Y}}_t(j)$
17:	end for