Do Joint Audio-Video Generation Models Understand Physics?
Abstract
Joint audio-video generation models are rapidly approaching professional production quality, raising a fundamental question: do these models truly understand audio-visual physics, or do they merely generate plausible sounds and frames that violate real-world physical consistency? To answer this question, we introduce AV-Phys Bench, the first comprehensive benchmark for evaluating physical commonsense in joint audio-video generation. AV-Phys Bench systematically tests joint audio-video generation models across three scene categories that probe how physical commonsense holds as the scene evolves: (a) Steady State, (b) Event Transition, and (c) Environment Transition. Each scene category investigates three physics-grounded subcategories that reflect real-world scenes, along with an additional Anti-AV-Physics subcategory, where prompts deliberately violate audio-visual physics to probe whether models possess generative physics knowledge or merely encode physically consistent priors. Each generation is scored along five dimensions: semantic adherence and physical commonsense within each modality, and cross-modal physical commonsense, which tests whether the visual and audio streams agree on the same physical event. Across three proprietary and four open-source models, Seedance 2.0 leads on physical commonsense, with an overall pass rate of 0.660. The gap to open-source models remains pronounced; every model degrades by up to 67% on event-driven and environment-driven transitions, and proprietary leaders collapse by 45–69% on Anti-AV-Physics prompts. Beyond human evaluation, we introduce AV-Phys Agent, a ReAct-style agentic evaluator that pairs a multimodal language model with deterministic acoustic measurement tools and ranks generators in close alignment with human ratings, enabling scalable physical-commonsense evaluation without further human annotation. AV-Phys Bench identifies cross-modal physical consistency and transition-driven scene dynamics as the open frontier for joint audio-video generation. We release AV-Phys Bench and AV-Phys Agent to facilitate future research on joint audio-video generation and evaluation.