UNIVERSE: Unified Video Action Models for Autonomous Driving with Flexible Mask-Modulated Modality Generation
Mengmeng Liu ⋅ Diankun Zhang ⋅ Jiuming Liu ⋅ Jianfeng Cui ⋅ Hongwei Xie ⋅ Guang Chen ⋅ Hangjun Ye ⋅ Francesco Nex ⋅ Hao Cheng ⋅ Michael Yang
Abstract
World Action Models have recently emerged as a promising paradigm for improving action generalization in autonomous driving by leveraging future video prediction as dense supervision for scene dynamics and temporal causality. However, it remains unclear how video modeling should be integrated with action generation to maximally transfer these benefits. Existing cascaded or dual-Diffusion Transformer architectures decouple video imagination from trajectory prediction, limiting knowledge transfer: the action model may still overfit dataset-specific driving priors, while the video model only indirectly regularizes planning. In this paper, we propose UNIVERSE, a unified video-action model built upon a single mask-modulated Diffusion Transformer. By co-training future video latents and ego-trajectory tokens within shared generative parameters, UNIVERSE allows dense video supervision to directly shape trajectory denoising, leading to stronger cross-domain action generalization. To ensure causal validity and efficient deployment, we introduce a Modality-Decoupling Visibility Mask, which shares historical context across modalities while blocking mutual attention between future video and trajectory tokens. This prevents future-modality dependency and enables trajectory-only inference by removing future-video denoising at test time, achieving a $4.3\times$ speedup over joint video--trajectory rollout while maintaining comparable planning accuracy. The same model also supports video-only and joint video--trajectory rollouts. Experiments show that UNIVERSE achieves 91.0 PDMS on NAVSIM and strong zero-shot transfer to nuScenes and Bench2Drive without fine-tuning, while ablations confirm the importance of single-DiT unification, video co-training, and mask-based modality decoupling.
Chat is not available.
Successful Page Load