Structured Identity Mapping Learning As a Model for Compositional Generalization in Generative Models

Multi-modal generative models demonstrate complex concept learning dynamics, initially learning individual concepts and later recombining them in novel ways not present in the training data. Despite the practical importance of understanding the causal mechanisms underlying these learning dynamics, our theoretical understanding remains limited. In this work, we aim to bridge this gap by systematically analyzing the learning dynamics of a simplified model: a one-hidden-layer network learning the identity map, with a training set composed of Gaussian point clouds non-uniformly distributed in space. We argue that a simple yet describe model of multi-modal generative model is the task of learning identity mapping.