Timezone: »

On Memorization in Probabilistic Deep Generative Models
Gerrit van den Burg · Chris Williams

Tue Dec 07 08:30 AM -- 10:00 AM (PST) @

Recent advances in deep generative models have led to impressive results in a variety of application domains. Motivated by the possibility that deep learning models might memorize part of the input data, there have been increased efforts to understand how memorization arises. In this work, we extend a recently proposed measure of memorization for supervised learning (Feldman, 2019) to the unsupervised density estimation problem and adapt it to be more computationally efficient. Next, we present a study that demonstrates how memorization can occur in probabilistic deep generative models such as variational autoencoders. This reveals that the form of memorization to which these models are susceptible differs fundamentally from mode collapse and overfitting. Furthermore, we show that the proposed memorization score measures a phenomenon that is not captured by commonly-used nearest neighbor tests. Finally, we discuss several strategies that can be used to limit memorization in practice. Our work thus provides a framework for understanding problematic memorization in probabilistic generative models.

Author Information

Gerrit van den Burg (N/A)

Gerrit van den Burg is a machine learning scientist based in London. He recently completed a postdoc at the Alan Turing Institute, the UK's national institute for data science and AI. Gerrit obtained his PhD from the Erasmus University Rotterdam in the Netherlands, and holds Master degrees in applied physics and econometrics. His research interests cover a number of machine learning topics, including kernel methods, time series analysis, automated data wrangling, and generative models. Gerrit's work has been published in JMLR, NeurIPS, DMKD, and AISTATS, and has been made available in various software packages for Python and R. You can find out more about his work at: https://gertjan.dev.

Chris Williams (University of Edinburgh)

More from the Same Authors