Hand-Object Interaction Image Generation

Hezhen Hu · Weilun Wang · Wengang Zhou · Houqiang Li

Hall J #636

Keywords: [ hand-object interaction ] [ complex occulsion ] [ Image Generation ]

[ Abstract ]
[ Paper [ Poster [ OpenReview
Wed 30 Nov 2 p.m. PST — 4 p.m. PST


In this work, we are dedicated to a new task, i.e., hand-object interaction image generation, which aims to conditionally generate the hand-object image under the given hand, object and their interaction status. This task is challenging and research-worthy in many potential application scenarios, such as AR/VR games and online shopping, etc. To address this problem, we propose a novel HOGAN framework, which utilizes the expressive model-aware hand-object representation and leverages its inherent topology to build the unified surface space. In this space, we explicitly consider the complex self- and mutual occlusion during interaction. During final image synthesis, we consider different characteristics of hand and object and generate the target image in a split-and-combine manner. For evaluation, we build a comprehensive protocol to access both the fidelity and structure preservation of the generated image. Extensive experiments on two large-scale datasets, i.e., HO3Dv3 and DexYCB, demonstrate the effectiveness and superiority of our framework both quantitatively and qualitatively. The code will be available at

Chat is not available.