Timezone: »

 
Poster
Auto Learning Attention
Benteng Ma · Jing Zhang · Yong Xia · Dacheng Tao

Thu Dec 10 09:00 PM -- 11:00 PM (PST) @ Poster Session 6 #1780
in Poster Session 7 »

Attention modules have been demonstrated effective in strengthening the representation ability of a neural network via reweighting spatial or channel features or stacking both operations sequentially. However, designing the structures of different attention operations requires a bulk of computation and extensive expertise. In this paper, we devise an Auto Learning Attention (AutoLA) method, which is the first attempt on automatic attention design. Specifically, we define a novel attention module named high order group attention (HOGA) as a directed acyclic graph (DAG) where each group represents a node, and each edge represents an operation of heterogeneous attentions. A typical HOGA architecture can be searched automatically via the differential AutoLA method within 1 GPU day using the ResNet-20 backbone on CIFAR10. Further, the searched attention module can generalize to various backbones as a plug-and-play component and outperforms popular manually designed channel and spatial attentions for many vision tasks, including image classification on CIFAR100 and ImageNet, object detection and human keypoint detection on COCO dataset. The code will be released.

Keywords: Representation Learning · Relational Learning · Algorithms

Author Information

Benteng Ma (Northwestern Polytechnical University)
Jing Zhang (The University of Sydney)
Yong Xia (Northwestern Polytechnical University, Research & Development Institute of Northwestern Polytechnical University in Shenzhen)
Dacheng Tao (University of Sydney)

More from the Same Authors