Timezone: »

 
Poster
Generative Occupancy Fields for 3D Surface-Aware Image Synthesis
Xudong XU · Xingang Pan · Dahua Lin · Bo Dai

Wed Dec 08 12:30 AM -- 02:00 AM (PST) @

The advent of generative radiance fields has significantly promoted the development of 3D-aware image synthesis. The cumulative rendering process in radiance fields makes training these generative models much easier since gradients are distributed over the entire volume, but leads to diffused object surfaces. In the meantime, compared to radiance fields occupancy representations could inherently ensure deterministic surfaces. However, if we directly apply occupancy representations to generative models, during training they will only receive sparse gradients located on object surfaces and eventually suffer from the convergence problem. In this paper, we propose Generative Occupancy Fields (GOF), a novel model based on generative radiance fields that can learn compact object surfaces without impeding its training convergence. The key insight of GOF is a dedicated transition from the cumulative rendering in radiance fields to rendering with only the surface points as the learned surface gets more and more accurate. In this way, GOF combines the merits of two representations in a unified framework. In practice, the training-time transition of start from radiance fields and march to occupancy representations is achieved in GOF by gradually shrinking the sampling region in its rendering process from the entire volume to a minimal neighboring region around the surface. Through comprehensive experiments on multiple datasets, we demonstrate that GOF can synthesize high-quality images with 3D consistency and simultaneously learn compact and smooth object surfaces. Our code is available at https://github.com/SheldonTsui/GOF_NeurIPS2021.

Author Information

Xudong XU (The Chinese University of Hong Kong)
Xingang Pan (Max Planck Institute for Informatics)
Dahua Lin (The Chinese University of Hong Kong)
Bo Dai (Shanghai AI Lab)

More from the Same Authors

  • 2022 Poster: Audio-Driven Co-Speech Gesture Video Generation »
    Xian Liu · Qianyi Wu · Hang Zhou · Yuanqi Du · Wayne Wu · Dahua Lin · Ziwei Liu
  • 2022 : MOPA: a Minimalist Off-Policy Approach to Safe-RL »
    Hao Sun · Ziping Xu · Zhenghao Peng · Meng Fang · Bo Dai · Bolei Zhou
  • 2022 : Factor Investing with a Deep Multi-Factor Model »
    Zikai Wei · Bo Dai · Dahua Lin
  • 2022 Spotlight: Lightning Talks 4B-4 »
    Chenze Shao · Ziyue Jiang · Wenguan Wang · Zeeshan Khan · Yichong Leng · Zehao Yu · Yuxiang Yang · Xian Liu · James Liang · Songyou Peng · Junjie Yang · Yang Feng · C.V. Jawahar · Qianyi Wu · Zehua Chen · Zhe Su · Hang Zhou · Yufei Xu · Junliang Guo · Dongfang Liu · Michael Niemeyer · Zhou Zhao · Makarand Tapaswi · Haohe Liu · Jing Zhang · Yuanqi Du · Torsten Sattler · Qian Yang · Yi Ren · Long Lan · Wayne Wu · Andreas Geiger · Jiawei Chen · Jinglin Liu · Dacheng Tao · Xu Tan · Dahua Lin · Danilo Mandic · Ziwei Liu · 振辉 叶 · Lei He · Xiangyang Li · Tao Qin · sheng zhao · Tie-Yan Liu
  • 2022 Spotlight: Audio-Driven Co-Speech Gesture Video Generation »
    Xian Liu · Qianyi Wu · Hang Zhou · Yuanqi Du · Wayne Wu · Dahua Lin · Ziwei Liu
  • 2022 Poster: Semi-Supervised Semantic Segmentation via Gentle Teaching Assistant »
    Ying Jin · Jiaqi Wang · Dahua Lin
  • 2022 Poster: Improving GANs with A Dynamic Discriminator »
    Ceyuan Yang · Yujun Shen · Yinghao Xu · Deli Zhao · Bo Dai · Bolei Zhou
  • 2021 Poster: A Shading-Guided Generative Implicit Model for Shape-Accurate 3D-Aware Image Synthesis »
    Xingang Pan · Xudong XU · Chen Change Loy · Christian Theobalt · Bo Dai
  • 2021 Poster: Balanced Chamfer Distance as a Comprehensive Metric for Point Cloud Completion »
    Tong Wu · Liang Pan · Junzhe Zhang · Tai WANG · Ziwei Liu · Dahua Lin
  • 2021 Poster: Few-Shot Object Detection via Association and DIscrimination »
    Yuhang Cao · Jiaqi Wang · Ying Jin · Tong Wu · Kai Chen · Ziwei Liu · Dahua Lin
  • 2021 Poster: Deceive D: Adaptive Pseudo Augmentation for GAN Training with Limited Data »
    Liming Jiang · Bo Dai · Wayne Wu · Chen Change Loy
  • 2019 : Poster and Coffee Break 2 »
    Karol Hausman · Kefan Dong · Ken Goldberg · Lihong Li · Lin Yang · Lingxiao Wang · Lior Shani · Liwei Wang · Loren Amdahl-Culleton · Lucas Cassano · Marc Dymetman · Marc Bellemare · Marcin Tomczak · Margarita Castro · Marius Kloft · Marius-Constantin Dinu · Markus Holzleitner · Martha White · Mengdi Wang · Michael Jordan · Mihailo Jovanovic · Ming Yu · Minshuo Chen · Moonkyung Ryu · Muhammad Zaheer · Naman Agarwal · Nan Jiang · Niao He · Nikolaus Yasui · Nikos Karampatziakis · Nino Vieillard · Ofir Nachum · Olivier Pietquin · Ozan Sener · Pan Xu · Parameswaran Kamalaruban · Paul Mineiro · Paul Rolland · Philip Amortila · Pierre-Luc Bacon · Prakash Panangaden · Qi Cai · Qiang Liu · Quanquan Gu · Raihan Seraj · Richard Sutton · Rick Valenzano · Robert Dadashi · Rodrigo Toro Icarte · Roshan Shariff · Roy Fox · Ruosong Wang · Saeed Ghadimi · Samuel Sokota · Sean Sinclair · Sepp Hochreiter · Sergey Levine · Sergio Valcarcel Macua · Sham Kakade · Shangtong Zhang · Sheila McIlraith · Shie Mannor · Shimon Whiteson · Shuai Li · Shuang Qiu · Wai Lok Li · Siddhartha Banerjee · Sitao Luan · Tamer Basar · Thinh Doan · Tianhe Yu · Tianyi Liu · Tom Zahavy · Toryn Klassen · Tuo Zhao · Vicenç Gómez · Vincent Liu · Volkan Cevher · Wesley Suttle · Xiao-Wen Chang · Xiaohan Wei · Xiaotong Liu · Xingguo Li · Xinyi Chen · Xingyou Song · Yao Liu · YiDing Jiang · Yihao Feng · Yilun Du · Yinlam Chow · Yinyu Ye · Yishay Mansour · · Yonathan Efroni · Yongxin Chen · Yuanhao Wang · Bo Dai · Chen-Yu Wei · Harsh Shrivastava · Hongyang Zhang · Qinqing Zheng · SIDDHARTHA SATPATHI · Xueqing Liu · Andreu Vall
  • 2019 : Poster and Coffee Break 1 »
    Aaron Sidford · Aditya Mahajan · Alejandro Ribeiro · Alex Lewandowski · Ali H Sayed · Ambuj Tewari · Angelika Steger · Anima Anandkumar · Asier Mujika · Hilbert J Kappen · Bolei Zhou · Byron Boots · Chelsea Finn · Chen-Yu Wei · Chi Jin · Ching-An Cheng · Christina Yu · Clement Gehring · Craig Boutilier · Dahua Lin · Daniel McNamee · Daniel Russo · David Brandfonbrener · Denny Zhou · Devesh Jha · Diego Romeres · Doina Precup · Dominik Thalmeier · Eduard Gorbunov · Elad Hazan · Elena Smirnova · Elvis Dohmatob · Emma Brunskill · Enrique Munoz de Cote · Ethan Waldie · Florian Meier · Florian Schaefer · Ge Liu · Gergely Neu · Haim Kaplan · Hao Sun · Hengshuai Yao · Jalaj Bhandari · James A Preiss · Jayakumar Subramanian · Jiajin Li · Jieping Ye · Jimmy Smith · Joan Bas Serrano · Joan Bruna · John Langford · Jonathan Lee · Jose A. Arjona-Medina · Kaiqing Zhang · Karan Singh · Yuping Luo · Zafarali Ahmed · Zaiwei Chen · Zhaoran Wang · Zhizhong Li · Zhuoran Yang · Ziping Xu · Ziyang Tang · Yi Mao · David Brandfonbrener · Shirli Di-Castro · Riashat Islam · Zuyue Fu · Abhishek Naik · Saurabh Kumar · Benjamin Petit · Angeliki Kamoutsi · Simone Totaro · Arvind Raghunathan · Rui Wu · Donghwan Lee · Dongsheng Ding · Alec Koppel · Hao Sun · Christian Tjandraatmadja · Mahdi Karami · Jincheng Mei · Chenjun Xiao · Junfeng Wen · Zichen Zhang · Ross Goroshin · Mohammad Pezeshki · Jiaqi Zhai · Philip Amortila · Shuo Huang · Mariya Vasileva · El houcine Bergou · Adel Ahmadyan · Haoran Sun · Sheng Zhang · Lukas Gruber · Yuanhao Wang · Tetiana Parshakova
  • 2019 Poster: Policy Continuation with Hindsight Inverse Dynamics »
    Hao Sun · Zhizhong Li · Xiaotong Liu · Bolei Zhou · Dahua Lin
  • 2019 Spotlight: Policy Continuation with Hindsight Inverse Dynamics »
    Hao Sun · Zhizhong Li · Xiaotong Liu · Bolei Zhou · Dahua Lin
  • 2018 Poster: A Neural Compositional Paradigm for Image Captioning »
    Bo Dai · Sanja Fidler · Dahua Lin
  • 2018 Poster: Trajectory Convolution for Action Recognition »
    Yue Zhao · Yuanjun Xiong · Dahua Lin
  • 2017 Poster: Contrastive Learning for Image Captioning »
    Bo Dai · Dahua Lin
  • 2013 Poster: Online Learning of Nonparametric Mixture Models via Sequential Variational Approximation »
    Dahua Lin
  • 2012 Poster: Coupling Nonparametric Mixtures via Latent Dirichlet Processes »
    Dahua Lin · John Fisher III
  • 2010 Oral: Construction of Dependent Dirichlet Processes based on Poisson Processes »
    Dahua Lin · Eric Grimson · John Fisher III
  • 2010 Poster: Construction of Dependent Dirichlet Processes based on Poisson Processes »
    Dahua Lin · Eric Grimson · John Fisher III