XFactors: Disentangled Information Bottleneck via Contrastive Supervision

Modern deep networks often learn features predictive of semantic attributes, but these factors are typically distributed across the representation rather than exposed as stable, addressable, testable variables. We study a practical version of disentanglement in which only a subset of factors is known, annotated, or actionable, the rest of the variation being unannotated, irrelevant to the downstream question, or too numerous to isolate exhaustively. We introduce ***XFactors***, a weakly-supervised VAE that makes selected factors directly interpretable through an explicit latent interface: the representation is decomposed into factor-specific subspaces $\mathcal{T}_1,\ldots,\mathcal{T}_K$ and a residual subspace $\mathcal{S}$, contrastive objectives align each selected factor with its assigned $\mathcal{T}_i$, and reconstruction and KL regularization retain non-targeted variation in $\mathcal{S}$ subject to the VAE bottleneck. The non-adversarial objective scales to multiple target factors by assigning one contrastive signal per block. With constant hyperparameters across benchmarks, \textsc{XFactors} obtains strong disentanglement scores, supports factor swapping by latent replacement, scales with residual capacity, and provides qualitative proof-of-concept results on real-world CelebA and JUMP Cell Painting data, where known biological and technical variables become directly interpretable