Bi-Lipschitz Factor Flows: Regularizing the Latent Geometry of Flow-Matching Decoders

Flow matching in its classical form is a generative model that does not require a latent representation. However, the analysis, interpolation and manipulation of factors in data require one, and its geometry is important. Indeed, in an autoencoder setting, a direction the decoder crushes can be moved without changing the output, while an over-expanded one can turn a small edit into an uncontrolled change after decoding. We ask when a flow-matching decoder, whose output is the endpoint of an ODE, is bi-Lipschitz in its latent code. The two sides of the bi-Lipschitz inequality behave differently. For the expansive bound (i.e. the upper Lipschitz bound) we show that enforcing local conditions on the velocity field suffices. For the contractive bound (i.e. the lower Lipschitz bound), we give a counterexample showing that local control of the velocity field does not prevent collapse at the endpoint. This is how we build BFF (Bi-Lipschitz Factor Flows). On two synthetic benchmarks and on Tabula Muris Senis, BFF reduces empirical decoder distortion by 29-40% with comparable reconstruction and stabilizes the recovery of a sub-dominant factor present in the data and never shown to the model.