ManipulationRAG: Retrieval-Augmented Fine-Grained Manipulation of Object Functional Parts

Modeling fine-grained human manipulation of object functional parts is essential for achieving human-like dexterous manipulation in virtual reality and robotics. Despite recent advances, data-driven generation methods remain largely constrained by patterns learned from limited training data, rendering them prone to hallucinating task-inconsistent manipulations when encountering samples beyond the training distribution. To address this challenge, we propose ManipulationRAG, a framework that augments an internal generative model with externally retrieved, task-relevant manipulation knowledge. Specifically, we design a hand-centric manipulation taxonomy and construct a structured knowledge base that organizes task descriptions, objects, manipulation types, and corresponding hand pose sequences, supported by a retrieval mechanism for accurate knowledge retrieval. The retrieved knowledge serves as an external manipulation-specific prior, guiding the diffusion model toward task-consistent and physically plausible manipulation synthesis. Extensive experiments demonstrate that our method outperforms existing methods and generalizes robustly to object manipulation beyond the original diffusion model's training domain.