Timezone: »

 
Poster
Rethinking Alignment in Video Super-Resolution Transformers
Shuwei Shi · Jinjin Gu · Liangbin Xie · Xintao Wang · Yujiu Yang · Chao Dong

@

The alignment of adjacent frames is considered an essential operation in video super-resolution (VSR). Advanced VSR models, including the latest VSR Transformers, are generally equipped with well-designed alignment modules. However, the progress of the self-attention mechanism may violate this common sense. In this paper, we rethink the role of alignment in VSR Transformers and make several counter-intuitive observations. Our experiments show that: (i) VSR Transformers can directly utilize multi-frame information from unaligned videos, and (ii) existing alignment methods are sometimes harmful to VSR Transformers. These observations indicate that we can further improve the performance of VSR Transformers simply by removing the alignment module and adopting a larger attention window. Nevertheless, such designs will dramatically increase the computational burden, and cannot deal with large motions. Therefore, we propose a new and efficient alignment method called patch alignment, which aligns image patches instead of pixels. VSR Transformers equipped with patch alignment could demonstrate state-of-the-art performance on multiple benchmarks. Our work provides valuable insights on how multi-frame information is used in VSR and how to select alignment methods for different networks/datasets. Codes and models will be released at https://github.com/XPixelGroup/RethinkVSRAlignment.

Keywords: transformer · alignment · self-attention · Video Super-Resolution
[ Paper]  [ Poster]  [ OpenReview] 

Author Information

Shuwei Shi (Tsinghua University)
Jinjin Gu (University of Sydney)
Liangbin Xie (Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Chinese Academy of Sciences)
Xintao Wang (Applied Research Center, Tencent PCG)
Yujiu Yang (Tsinghua University)
Chao Dong (Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Chinese Academy of Sciences)

More from the Same Authors