Algorithm 1.  Overall Pipeline of TrackRefine

Input: Video sequence $\mathcal{V}=\{I_t{\}}_{t=1}^T$, front-end instance segmenter $\mathsf{\Phi }$, memory bank $\mathcal{M}$, thresholds $\mathsf{\Theta }$ Output: Trajectory set $\mathcal{T}$ 1  Initialize $\mathcal{T}\leftarrow \mathrm{\varnothing }$ and $\mathcal{M}\leftarrow \mathrm{\varnothing }$ 2  for $t =1$ to $T$ do 3     Obtain frame-wise detections:

${\mathcal{D}}_t=\mathsf{\Phi }\left(I_t\right)=\left\{\right(b_i^t,m_i^t,c_i^t,s_i^t){\}}_{i=1}^{N_t}$

4     Suppress highly overlapped duplicate masks in ${\mathcal{D}}_t$ 5     Refine initial masks by local Fast GrabCut:

${\hat{\mathcal{D}}}_t=\mathrm{M}\mathrm{a}\mathrm{s}\mathrm{k}\mathrm{R}\mathrm{e}\mathrm{f}\mathrm{i}\mathrm{n}\mathrm{e}(I_t,{\mathcal{D}}_t)$

6     Extract appearance feature $f_i^{app,t}$, semantic feature $f_i^{sem,t}$, and shape feature $f_i^{shp,t}$ from ${\hat{\mathcal{D}}}_t$ 7     Fuse semantic and shape cues:

$f_i^{joi,t}=\mathrm{J}\mathrm{o}\mathrm{i}\mathrm{n}\mathrm{t}\mathrm{F}\mathrm{e}\mathrm{a}\mathrm{t}\mathrm{u}\mathrm{r}\mathrm{e}\mathrm{E}\mathrm{n}\mathrm{c}\mathrm{o}\mathrm{d}\mathrm{e}\left(f_i^{sem,t},f_i^{shp,t}\right)$

8     Construct multimodal feature:

${\mathcal{F}}_i^t=\left(f_i^{app,t},f_i^{joi,t}\right)$

9     Construct the multimodal feature set:

${\mathcal{F}}_t=\{{\mathcal{F}}_i^t{\}}_{i=1}^{N_t}$

10     Predict existing trajectory states using Kalman filtering and camera motion compensation 11   Partition trajectories into active, recently unmatched, and lost states 12       Perform progressive three-stage association:

${\mathcal{A}}_t,{\mathcal{U}}_t=\mathrm{T}\mathrm{h}\mathrm{r}\mathrm{e}\mathrm{e}\mathrm{S}\mathrm{t}\mathrm{a}\mathrm{g}\mathrm{e}\mathrm{A}\mathrm{s}\mathrm{s}\mathrm{o}\mathrm{c}\mathrm{i}\mathrm{a}\mathrm{t}\mathrm{e}(\mathcal{T},{\hat{\mathcal{D}}}_t,{\mathcal{F}}_t,\mathcal{M})$

13   for each matched pair $({\tau }_k,d_j)\in {\mathcal{A}}_t$ do 14           Update the state, box, mask, and identity of ${\tau }_k$ using $d_j$ 15           Estimate the observation quality $Q_j^t$ 16             Selectively update short-term memory and long-term prototypes:

${\mathcal{M}}_k\leftarrow \mathrm{Q}\mathrm{u}\mathrm{a}\mathrm{l}\mathrm{i}\mathrm{t}\mathrm{y}\mathrm{A}\mathrm{w}\mathrm{a}\mathrm{r}\mathrm{e}\mathrm{U}\mathrm{p}\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{e}({\mathcal{M}}_k,d_j,{\mathcal{F}}_j^t,Q_j^t)$

17    end for 18    Mark unmatched trajectories as occluded or lost according to missing duration 19    Initialize new tentative trajectories for unmatched detections in ${\mathcal{U}}_t$ 20    Remove expired trajectories from $\mathcal{T}$ 21 end for 22  $\mathbf{r}\mathbf{e}\mathbf{t}\mathbf{u}\mathbf{r}\mathbf{n} \mathcal{T}$