Semantic-level Exploration for Multi-Agent Reinforcement Learning

Multi-Agent Reinforcement Learning (MARL) faces significant exploration challenges due to exponentially growing joint state-action spaces. Existing exploration methods operate directly in raw state-action spaces, which is inefficient and fails to exploit inherent semantic structure. This paper introduces semantic space into multi-agent exploration. We theoretically establish that the observation-action space can be partitioned into discrete semantic prototypes, providing a principled foundation for transferring exploration statistics across semantically similar situations. Building on this theory, we propose a semantic-level exploration mechanism that first compresses the high-dimensional observation–action space into discrete semantic prototypes via vector quantization, and then applies sliding-window count-based bonuses to enable efficient statistical transfer across semantically similar situations. Our approach is versatile and can be seamlessly integrated with existing value-based MARL frameworks. Extensive experiments demonstrate that our method outperforms state-of-the-art baselines across diverse multi-agent benchmarks in terms of both effectiveness and training efficiency.