Timezone: »

 
Poster
Emergent Correspondence from Image Diffusion
Luming Tang · Menglin Jia · Qianqian Wang · Cheng Perng Phoo · Bharath Hariharan

Thu Dec 14 03:00 PM -- 05:00 PM (PST) @ Great Hall & Hall B1+B2 #220
in Poster Session 6 »
Event URL: https://diffusionfeatures.github.io/ »

Finding correspondences between images is a fundamental problem in computer vision. In this paper, we show that correspondence emerges in image diffusion models without any explicit supervision. We propose a simple strategy to extract this implicit knowledge out of diffusion networks as image features, namely DIffusion FeaTures (DIFT), and use them to establish correspondences between real images. Without any additional fine-tuning or supervision on the task-specific data or annotations, DIFT is able to outperform both weakly-supervised methods and competitive off-the-shelf features in identifying semantic, geometric, and temporal correspondences. Particularly for semantic correspondence, DIFT from Stable Diffusion is able to outperform DINO and OpenCLIP by 19 and 14 accuracy points respectively on the challenging SPair-71k benchmark. It even outperforms the state-of-the-art supervised methods on 9 out of 18 categories while remaining on par for the overall performance. Project page: https://diffusionfeatures.github.io.

Author Information

Luming Tang (Cornell University)
Menglin Jia (Cornell University)
Qianqian Wang (Cornell university)
Cheng Perng Phoo (Cornell University)
Bharath Hariharan (Cornell University)

More from the Same Authors