Efficient Video Prediction via Sparsely Conditioned Flow Matching

Davtyan, Aram; Sameni, Sepehr; Favaro, Paolo (2023). Efficient Video Prediction via Sparsely Conditioned Flow Matching. In: International Conference on Computer Vision.

[img] Text
Davtyan_Efficient_Video_Prediction_via_Sparsely_Conditioned_Flow_Matching_ICCV_2023_paper.pdf - Published Version
Restricted to registered users only
Available under License Publisher holds Copyright.

Download (3MB) | Request a copy

We introduce a novel generative model for video prediction based on latent flow matching, an efficient alternative to diffusion-based models. In contrast to prior work, we keep the high costs of modeling the past during training and inference at bay by conditioning only on a small random set of past frames at each integration step of the image generation process. Moreover, to enable the generation of high-resolution videos and to speed up the training, we work in the latent space of a pretrained VQGAN. Finally, we propose to approximate the initial condition of the flow ODE with the previous noisy frame. This allows to reduce the number of integration steps and hence, speed up the sampling at inference time. We call our model Random frame conditioned flow Integration for VidEo pRediction, or, in short, RIVER. We show that RIVER achieves superior or on par performance compared to prior work on common video prediction benchmarks, while requiring an order of magnitude fewer computational resources. Project website: https://araachie.github.io/river.

Item Type:

Conference or Workshop Item (Paper)

Division/Institute:

08 Faculty of Science > Institute of Computer Science (INF) > Computer Vision Group (CVG)
08 Faculty of Science > Institute of Computer Science (INF)
08 Faculty of Science > Institute of Computer Science (INF) > Computer Graphics Group (CGG)

UniBE Contributor:

Davtyan, Aram, Sameni, Sepehr, Favaro, Paolo

Subjects:

000 Computer science, knowledge & systems
600 Technology > 620 Engineering
500 Science > 510 Mathematics
000 Computer science, knowledge & systems > 050 Magazines, journals & serials

Language:

English

Submitter:

Llukman Cerkezi

Date Deposited:

18 Apr 2024 14:26

Last Modified:

18 Apr 2024 14:26

BORIS DOI:

10.48350/196071

URI:

https://boris.unibe.ch/id/eprint/196071

Actions (login required)

Edit item Edit item
Provide Feedback