Timezone: »

 
Poster
LION: Latent Point Diffusion Models for 3D Shape Generation
xiaohui zeng · Arash Vahdat · Francis Williams · Zan Gojcic · Or Litany · Sanja Fidler · Karsten Kreis

Tue Nov 29 09:00 AM -- 11:00 AM (PST) @ Hall J #625
in Poster Session 1 »

Denoising diffusion models (DDMs) have shown promising results in 3D point cloud synthesis. To advance 3D DDMs and make them useful for digital artists, we require (i) high generation quality, (ii) flexibility for manipulation and applications such as conditional synthesis and shape interpolation, and (iii) the ability to output smooth surfaces or meshes. To this end, we introduce the hierarchical Latent Point Diffusion Model (LION) for 3D shape generation. LION is set up as a variational autoencoder (VAE) with a hierarchical latent space that combines a global shape latent representation with a point-structured latent space. For generation, we train two hierarchical DDMs in these latent spaces. The hierarchical VAE approach boosts performance compared to DDMs that operate on point clouds directly, while the point-structured latents are still ideally suited for DDM-based modeling. Experimentally, LION achieves state-of-the-art generation performance on multiple ShapeNet benchmarks. Furthermore, our VAE framework allows us to easily use LION for different relevant tasks: LION excels at multimodal shape denoising and voxel-conditioned synthesis, and it can be adapted for text- and image-driven 3D generation. We also demonstrate shape autoencoding and latent shape interpolation, and we augment LION with modern surface reconstruction techniques to generate smooth 3D meshes. We hope that LION provides a powerful tool for artists working with 3D shapes due to its high-quality generation, flexibility, and surface reconstruction. Project page and code: https://nv-tlabs.github.io/LION.

Keywords: variational autoencoder · diffusion models · Generative Learning · Denoising Diffusion · 3D Shape Synthesis · Point Cloud Generation

Author Information

xiaohui zeng (university of Toronto)
Arash Vahdat (NVIDIA Research)
Francis Williams (NVIDIA)
Francis Williams

I am a [research scientist at NVIDIA](https://nv-tlabs.github.io/) in NYC working at the intersection of computer vision, machine learning, and computer graphics. My research is a mix of theory and application, aiming to solve practical problems in elegant ways. In particular, I’m very interested in 3d shape representations which can enable deep learning on “real-world” geometric datasets which are often noisy, unlabeled, and consisting of very large inputs. I completed my PhD from NYU in 2021 where I worked in the [Math and Data Group](https://mad.cds.nyu.edu/) and the [Geometric Computing Lab](https://cims.nyu.edu/gcl/). My advisors were [Joan Bruna](https://cims.nyu.edu/~bruna/) and [Denis Zorin](https://cims.nyu.edu/gcl/denis.html). In addition to doing research, I am the creator and maintainer of several [open source projects](https://github.com/fwilliams). These include [NumpyEigen](https://github.com/fwilliams/numpyeigen), [Point Cloud Utils](https://github.com/fwilliams/point-cloud-utils), and [FML](https://github.com/fwilliams/fml). I’m currently looking for motivated interns to work with. Please reach out to me if you would like to chat about potential collaborations at NVIDIA!

Zan Gojcic (NVIDIA)
Or Litany (NVIDIA)
Sanja Fidler (TTI at Chicago)
Karsten Kreis (NVIDIA)

More from the Same Authors