Timezone: »

 
Spotlight Poster
Complexity Matters: Rethinking the Latent Space for Generative Modeling
Tianyang Hu · Fei Chen · Haonan Wang · Jiawei Li · Wenjia Wang · Jiacheng Sun · Zhenguo Li

Tue Dec 12 03:15 PM -- 05:15 PM (PST) @ Great Hall & Hall B1+B2 #543
in Poster Session 2 »

In generative modeling, numerous successful approaches leverage a low-dimensional latent space, e.g., Stable Diffusion models the latent space induced by an encoder and generates images through a paired decoder. Although the selection of the latent space is empirically pivotal, determining the optimal choice and the process of identifying it remain unclear. In this study, we aim to shed light on this under-explored topic by rethinking the latent space from the perspective of model complexity. Our investigation starts with the classic generative adversarial networks (GANs). Inspired by the GAN training objective, we propose a novel "distance" between the latent and data distributions, whose minimization coincides with that of the generator complexity. The minimizer of this distance is characterized as the optimal data-dependent latent that most effectively capitalizes on the generator's capacity. Then, we consider parameterizing such a latent distribution by an encoder network and propose a two-stage training strategy called Decoupled Autoencoder (DAE), where the encoder is only updated in the first stage with an auxiliary decoder and then frozen in the second stage while the actual decoder is being trained. DAE can improve the latent distribution and as a result, improve the generative performance. Our theoretical analyses are corroborated by comprehensive experiments on various models such as VQGAN and Diffusion Transformer, where our modifications yield significant improvements in sample quality with decreased model complexity.

Author Information

Tianyang Hu (Huawei Noah’s Ark Lab)
Fei Chen (Huawei Noah's Ark Lab)
Haonan Wang (national university of singaore, National University of Singapore)
Jiawei Li (Huawei Technologies Ltd.)
Wenjia Wang (SAMSI)
Jiacheng Sun (Huawei Technologies Co., Ltd)
Zhenguo Li (Noah's Ark Lab, Huawei Tech Investment Co Ltd)

More from the Same Authors