Controllable Video Generation with Sparse Trajectories
Publikation: Bidrag til tidsskrift › Konferenceartikel › Forskning › fagfællebedømt
Video generation and manipulation is an important yet challenging task in computer vision. Existing methods usually lack ways to explicitly control the synthesized motion. In this work, we present a conditional video generation model that allows detailed control over the motion of the generated video. Given the first frame and sparse motion trajectories specified by users, our model can synthesize a video with corresponding appearance and motion. We propose to combine the advantage of copying pixels from the given frame and hallucinating the lightness difference from scratch which help generate sharp video while keeping the model robust to occlusion and lightness change. We also propose a training paradigm that calculate trajectories from video clips, which eliminated the need of annotated training data. Experiments on several standard benchmarks demonstrate that our approach can generate realistic videos comparable to state-of-the-art video generation and video prediction methods while the motion of the generated videos can correspond well with user input.
Originalsprog | Engelsk |
---|---|
Tidsskrift | Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition |
Sider (fra-til) | 7854-7863 |
Antal sider | 10 |
ISSN | 1063-6919 |
DOI | |
Status | Udgivet - 14 dec. 2018 |
Eksternt udgivet | Ja |
Begivenhed | 31st Meeting of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2018 - Salt Lake City, USA Varighed: 18 jun. 2018 → 22 jun. 2018 |
Konference
Konference | 31st Meeting of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2018 |
---|---|
Land | USA |
By | Salt Lake City |
Periode | 18/06/2018 → 22/06/2018 |
Bibliografisk note
Publisher Copyright:
© 2018 IEEE.
ID: 301825079