Timezone: »
Recently, the accuracy of image-text matching has been greatly improved by multimodal pretrained models, all of which are trained on millions or billions of paired images and texts. Different from them, this paper studies a new scenario as unpaired image-text matching, in which paired images and texts are assumed to be unavailable during model training. To deal with this, we propose a simple yet effective method namely Multimodal Aligned Conceptual Knowledge (MACK), which is inspired by the knowledge use in human brain. It can be directly used as general knowledge to correlate images and texts even without model training, or further fine-tuned based on unpaired images and texts to better generalize to certain datasets. In addition, we extend it as a re-ranking method, which can be easily combined with existing image-text matching models to substantially improve their performance.
Author Information
Yan Huang (CRIPAC, CASIA)
Yuming Wang (Institute of automation, Chinese academy of science, Chinese Academy of Sciences)
Yunan Zeng (Institute of automation, Chinese academy of science, Chinese Academy of Sciences)
Liang Wang (NLPR, China)
Related Events (a corresponding poster, oral, or spotlight)
-
2022 Poster: MACK: Multimodal Aligned Conceptual Knowledge for Unpaired Image-text Matching »
Wed. Nov 30th 05:00 -- 07:00 PM Room Hall J #136
More from the Same Authors
-
2023 Poster: Frequency-Enhanced Data Augmentation for Vision-and-Language Navigation »
Keji He · Chenyang Si · Zhihe Lu · Yan Huang · Liang Wang · Xinchao Wang -
2023 Poster: OneNet: Enhancing Time Series Forecasting Models under Concept Drift by Online Ensembling »
yifan zhang · Qingsong Wen · xue wang · Weiqi Chen · Liang Sun · Zhang Zhang · Liang Wang · Rong Jin · Tieniu Tan -
2023 Poster: A Comparative Study of Graph Structure Learning: Benchmark and Analysis »
Zhixun Li · Liang Wang · Xin Sun · Yifan Luo · Yanqiao Zhu · Dingshuo Chen · Yingtao Luo · Xiangxin Zhou · Qiang Liu · Shu Wu · Liang Wang · Jeffrey Yu -
2023 Poster: Uncovering Neural Scaling Law in Molecular Representation Learning »
Dingshuo Chen · Yanqiao Zhu · Jieyu Zhang · Yuanqi Du · Zhixun Li · Qiang Liu · Shu Wu · Liang Wang -
2021 Poster: Landmark-RxR: Solving Vision-and-Language Navigation with Fine-Grained Alignment Supervision »
Keji He · Yan Huang · Qi Wu · Jianhua Yang · Dong An · Shuanglin Sima · Liang Wang -
2020 Poster: Unfolding the Alternating Optimization for Blind Super Resolution »
zhengxiong luo · Yan Huang · Shang Li · Liang Wang · Tieniu Tan -
2015 Poster: Bidirectional Recurrent Convolutional Networks for Multi-Frame Super-Resolution »
Yan Huang · Wei Wang · Liang Wang -
2013 Poster: Relevance Topic Model for Unstructured Social Group Activity Recognition »
Fang Zhao · Yongzhen Huang · Liang Wang · Tieniu Tan