Algorithm 1 Learning without forgetting

Start with:   ${\theta }_s$: shared parameters   ${\theta }_o$: task specific parameters for each old task   $X_n$, $Y_n$: training data and ground truth on the new task Initialize:   $Y_o\leftarrow CNN(X_n,{\theta }_s,{\theta }_o)$       // compute output of old tasks for new data   ${\theta }_n\leftarrow RANDINIT\left(\right|{\theta }_n\left|\right)$    // randomly initialize new parameters Train:   Define ${\widehat{Y}}_o\equiv CNN(X_n,{\theta }_s,{\theta }_o)$       // old task output   Define ${\widehat{Y}}_n\equiv CNN(X_n,{\theta }_s,{\theta }_n)$       // new task output   ${\theta }_s*,{\theta }_o*,{\theta }_n*\equiv \underset{{\theta }_s*,{\theta }_o*,{\theta }_n*}{argmin}({\lambda }_o{L}_{old}(Y_o,\widehat{Y_o})+L_{new}(Y_n,\widehat{Y_n})+R({\widehat{\theta }}_s+{\widehat{\theta }}_o+{\widehat{\theta }}_n)$