Timezone: »
We present a video generation model that accurately reproduces object motion, changes in camera viewpoint, and new content that arises over time. Existing video generation methods often fail to produce new content as a function of time while maintaining consistencies expected in real environments, such as plausible dynamics and object persistence. A common failure case is for content to never change due to over-reliance on inductive bias to provide temporal consistency, such as a single latent code that dictates content for the entire video. On the other extreme, without long-term consistency, generated videos may morph unrealistically between different scenes. To address these limitations, we prioritize the time axis by redesigning the temporal latent representation and learning long-term consistency from data by training on longer videos. We leverage a two-phase training strategy, where we separately train using longer videos at a low resolution and shorter videos at a high resolution. To evaluate the capabilities of our model, we introduce two new benchmark datasets with explicit focus on long-term temporal dynamics.
Author Information
Tim Brooks (UC Berkeley)
Janne Hellsten (NVIDIA)
Miika Aittala (NVIDIA)
Ting-Chun Wang (NVIDIA)
Timo Aila (NVIDIA)
Jaakko Lehtinen (Aalto University & NVIDIA)
Ming-Yu Liu (NVIDIA)
Alexei Efros (UC Berkeley)
Tero Karras (NVIDIA)
More from the Same Authors
-
2022 Poster: Elucidating the Design Space of Diffusion-Based Generative Models »
Tero Karras · Miika Aittala · Timo Aila · Samuli Laine -
2022 : Studying Bias in GANs through the Lens of Race »
Vongani Maluleke · Neerja Thakkar · Tim Brooks · Ethan Weber · Trevor Darrell · Alexei Efros · Angjoo Kanazawa · Devin Guillory -
2023 Poster: Diffusion Self-Guidance for Controllable Image Generation »
Dave Epstein · Allan Jabri · Ben Poole · Alexei Efros · Aleksander Holynski -
2023 Poster: Differentiable Blocks World: Qualitative 3D Decomposition by Rendering Primitives »
Tom Monnier · Jake Austin · Angjoo Kanazawa · Alexei Efros · Mathieu Aubry -
2022 Panel: Panel 5B-2: Training and Inference… & Elucidating the Design… »
Andy Shih · Tero Karras -
2022 Spotlight: Lightning Talks 3B-2 »
Yu Huang · Tero Karras · Maxim Kodryan · Shiau Hong Lim · Shudong Huang · Ziyu Wang · Siqiao Xue · ILYAS MALIK · Ekaterina Lobacheva · Miika Aittala · Hongjie Wu · Yuhao Zhou · Yingbin Liang · Xiaoming Shi · Jun Zhu · Maksim Nakhodnov · Timo Aila · Yazhou Ren · James Zhang · Longbo Huang · Dmitry Vetrov · Ivor Tsang · Hongyuan Mei · Samuli Laine · Zenglin Xu · Wentao Feng · Jiancheng Lv -
2022 Spotlight: Elucidating the Design Space of Diffusion-Based Generative Models »
Tero Karras · Miika Aittala · Timo Aila · Samuli Laine -
2022 Poster: Implicit Warping for Animation with Image Sets »
Arun Mallya · Ting-Chun Wang · Ming-Yu Liu -
2022 Poster: Implicit Neural Representations with Levels-of-Experts »
Zekun Hao · Arun Mallya · Serge Belongie · Ming-Yu Liu -
2022 Poster: Test-Time Training with Masked Autoencoders »
Yossi Gandelsman · Yu Sun · Xinlei Chen · Alexei Efros -
2022 Poster: Visual Prompting via Image Inpainting »
Amir Bar · Yossi Gandelsman · Trevor Darrell · Amir Globerson · Alexei Efros -
2021 Poster: Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis »
Tianchang Shen · Jun Gao · Kangxue Yin · Ming-Yu Liu · Sanja Fidler -
2021 Poster: Alias-Free Generative Adversarial Networks »
Tero Karras · Miika Aittala · Samuli Laine · Erik Härkönen · Janne Hellsten · Jaakko Lehtinen · Timo Aila -
2021 Poster: MarioNette: Self-Supervised Sprite Learning »
Dmitriy Smirnov · MICHAEL GHARBI · Matthew Fisher · Vitor Guizilini · Alexei Efros · Justin Solomon -
2021 Oral: Alias-Free Generative Adversarial Networks »
Tero Karras · Miika Aittala · Samuli Laine · Erik Härkönen · Janne Hellsten · Jaakko Lehtinen · Timo Aila -
2020 : Panel Discussion & Closing »
Yejin Choi · Alexei Efros · Chelsea Finn · Kristen Grauman · Quoc V Le · Yann LeCun · Ruslan Salakhutdinov · Eric Xing -
2020 : QA: Alexei Efros »
Alexei Efros -
2020 : Invited Talk: Alexei Efros »
Alexei Efros -
2020 Poster: Learning compositional functions via multiplicative weight updates »
Jeremy Bernstein · Jiawei Zhao · Markus Meister · Ming-Yu Liu · Anima Anandkumar · Yisong Yue -
2020 Poster: Space-Time Correspondence as a Contrastive Random Walk »
Allan Jabri · Andrew Owens · Alexei Efros -
2020 Oral: Space-Time Correspondence as a Contrastive Random Walk »
Allan Jabri · Andrew Owens · Alexei Efros -
2020 Poster: Training Generative Adversarial Networks with Limited Data »
Tero Karras · Miika Aittala · Janne Hellsten · Samuli Laine · Jaakko Lehtinen · Timo Aila -
2020 Poster: GANSpace: Discovering Interpretable GAN Controls »
Erik Härkönen · Aaron Hertzmann · Jaakko Lehtinen · Sylvain Paris -
2020 Oral: Training Generative Adversarial Networks with Limited Data »
Tero Karras · Miika Aittala · Janne Hellsten · Samuli Laine · Jaakko Lehtinen · Timo Aila -
2020 Poster: Swapping Autoencoder for Deep Image Manipulation »
Taesung Park · Jun-Yan Zhu · Oliver Wang · Jingwan Lu · Eli Shechtman · Alexei Efros · Richard Zhang -
2019 : Poster Presentations »
Rahul Mehta · Andrew Lampinen · Binghong Chen · Sergio Pascual-Diaz · Jordi Grau-Moya · Aldo Faisal · Jonathan Tompson · Yiren Lu · Khimya Khetarpal · Martin Klissarov · Pierre-Luc Bacon · Doina Precup · Thanard Kurutach · Aviv Tamar · Pieter Abbeel · Jinke He · Maximilian Igl · Shimon Whiteson · Wendelin Boehmer · Raphaël Marinier · Olivier Pietquin · Karol Hausman · Sergey Levine · Chelsea Finn · Tianhe Yu · Lisa Lee · Benjamin Eysenbach · Emilio Parisotto · Eric Xing · Ruslan Salakhutdinov · Hongyu Ren · Anima Anandkumar · Deepak Pathak · Christopher Lu · Trevor Darrell · Alexei Efros · Phillip Isola · Feng Liu · Bo Han · Gang Niu · Masashi Sugiyama · Saurabh Kumar · Janith Petangoda · Johan Ferret · James McClelland · Kara Liu · Animesh Garg · Robert Lange -
2019 : Oral Presentations »
Janith Petangoda · Sergio Pascual-Diaz · Jordi Grau-Moya · Raphaël Marinier · Olivier Pietquin · Alexei Efros · Phillip Isola · Trevor Darrell · Christopher Lu · Deepak Pathak · Johan Ferret -
2019 Poster: High-Quality Self-Supervised Deep Image Denoising »
Samuli Laine · Tero Karras · Jaakko Lehtinen · Timo Aila -
2019 Poster: Learning to Control Self-Assembling Morphologies: A Study of Generalization via Modularity »
Deepak Pathak · Christopher Lu · Trevor Darrell · Phillip Isola · Alexei Efros -
2019 Spotlight: Learning to Control Self-Assembling Morphologies: A Study of Generalization via Modularity »
Deepak Pathak · Christopher Lu · Trevor Darrell · Phillip Isola · Alexei Efros -
2019 Poster: Improved Precision and Recall Metric for Assessing Generative Models »
Tuomas Kynkäänniemi · Tero Karras · Samuli Laine · Jaakko Lehtinen · Timo Aila -
2017 : How to stop worrying and learn to love Nearest Neighbors »
Alexei Efros -
2017 Poster: Toward Multimodal Image-to-Image Translation »
Jun-Yan Zhu · Richard Zhang · Deepak Pathak · Trevor Darrell · Alexei Efros · Oliver Wang · Eli Shechtman -
2016 : What makes ImageNet good for Transfer Learning? »
Jacob MY Huh · Pulkit Agrawal · Alexei Efros