DexMirror: Real-to-Sim Scene Mirroring for Sim-to-Real Dexterous Manipulation

Dexterous manipulation requires large-scale robot interaction data, yet collecting real-world demonstrations is costly, while sim-to-real transfer remains challenging due to visual and geometric discrepancies. We present DexMirror, a unified real-to-sim-to-real framework that enables photorealistic simulation learning and reliable deployment for dexterous manipulation. Our method reconstructs real scenarios using a compositional 3D Gaussian Splatting (3DGS) representation with tri-level optimization, obtaining high-fidelity objects and background while preserving global scene consistency. A dexterous-hand-centric calibration pipeline, supported by an interactive online platform, further achieves millimeter-level robot–scene consistency for efficient Gaussian-simulation alignment. Built upon the reconstructed scenes, we train privileged policies in simulation and distill them into visuomotor policies operating on rendered 3DGS observations, combined with closed-loop execution for robust transfer. Experiments demonstrate accurate real-to-sim reconstruction (F1 score 0.981) and strong sim-to-real performance, achieving high success rates across six real-world dexterous manipulation tasks.