Figure 1.

Overview: Our approach predicts the next frame ${\mu }_t$ of a video autoregressively along with an additive correction ${\mathbf{y}}_0^t$ generated by a denoising process. Detailed model: Two convolutional RNNs (blue and red arrows) operate on a frame sequence ${\mathbf{x}}^{0:t-1}$ to predict the most likely next frame ${\mu }^t$ (blue box) and a context vector for a denoising diffusion model. The diffusion model is trained to model the scaled residual ${\mathbf{y}}_0^t=({\mathbf{x}}^t-{\mu }^t)/\sigma$ conditioned on the temporal context. At generation time, the generated residual is added to the next-frame estimate ${\mu }^t$ to generate the next frame as ${\mathbf{x}}^t={\mu }^t+\sigma {\mathbf{y}}_0^t$.