SONIC-O1: A Real-World Benchmark for Evaluating Multimodal Large Language Models on Audio-Video Understanding

Multimodal Large Language Models (MLLMs) are a major focus of recent AI research. However, most prior work focuses on static image understanding, while their ability to process audio–video data remains underexplored. This gap highlights the need for a high-quality benchmark to systematically evaluate MLLM performance in a real-world setting. We introduce SONIC-O1, a comprehensive, fully human-verified benchmark of $\approx$60 hours (231 clips) spanning 13 real-world conversational domains with 4,958 annotations and perceived demographic metadata. SONIC-O1 evaluates three capabilities: open-ended summarization, multiple-choice question (MCQ) answering, and temporal localization with supporting rationales (reasoning). Across closed- and open-source models, we find that the MCQ accuracy shows the smallest gap between model families, under their recommended settings, the best closed-source model outperforms the best open-source model by 22.6% on temporal localization. We further observe accuracy gaps of up to 21.4% on temporal localization across demographic groups, indicating diagnostic disparities in model behaviour. SONIC-O1 provides an open evaluation suite for temporally grounded and demographically informed multimodal understanding. We release SONIC-O1 for reproducibility and research: Dataset: https://huggingface.co/datasets/sonico1org/sonico1 , Code Repository: https://github.com/sonico1benchmark/sonico1. We fully comply with the double-blind review process. All code and data are anonymized in the submission.