Timezone: »
Auxiliary learning is a widely adopted practice in deep learning, which aims to improve the model performance on the primary task by exploiting the beneficial information in the auxiliary loss. Existing auxiliary learning methods only focus on balancing the auxiliary loss and the primary loss, ignoring the module-level auxiliary influence, i.e., an auxiliary loss will be beneficial for optimizing specific modules within the model but harmful to others, failing to make full use of auxiliary information. To tackle the problem, we propose a Module-Aware Optimization approach for Auxiliary Learning (MAOAL). The proposed approach considers the module-level influence through the learnable module-level auxiliary importance, i.e., the importance of each auxiliary loss to each module. Specifically, the proposed approach jointly optimizes the module-level auxiliary importance and the model parameters in a bi-level manner. In the lower optimization, the model parameters are optimized with the importance parameterized gradient, while in the upper optimization, the module-level auxiliary importance is updated with the implicit gradient from a small developing dataset. Extensive experiments show that our proposed MAOAL method consistently outperforms state-of-the-art baselines for different auxiliary losses on various datasets, demonstrating that our method can serve as a powerful generic tool for auxiliary learning.
Author Information
Hong Chen (Tsinghua University)
Xin Wang (Tsinghua University)
Yue Liu (Tsinghua University, Tsinghua University)
Yuwei Zhou (Tsinghua University, Tsinghua University)
Chaoyu Guan (Tsinghua University, Tsinghua University)
Wenwu Zhu (Tsinghua University)
More from the Same Authors
-
2022 Poster: Learning Invariant Graph Representations for Out-of-Distribution Generalization »
Haoyang Li · Ziwei Zhang · Xin Wang · Wenwu Zhu -
2022 Poster: Dynamic Graph Neural Networks Under Spatio-Temporal Distribution Shift »
Zeyang Zhang · Xin Wang · Ziwei Zhang · Haoyang Li · Zhou Qin · Wenwu Zhu -
2022 Poster: NAS-Bench-Graph: Benchmarking Graph Neural Architecture Search »
Yijian Qin · Ziwei Zhang · Xin Wang · Zeyang Zhang · Wenwu Zhu -
2023 Poster: Fused Gromov-Wasserstein Graph Mixup for Graph-level Classifications »
Xinyu Ma · Xu Chu · Yasha Wang · Yang Lin · Junfeng Zhao · Liantao Ma · Wenwu Zhu -
2023 Poster: Unsupervised Graph Neural Architecture Search with Disentangled Self-Supervision »
Zeyang Zhang · Xin Wang · Ziwei Zhang · Guangyao Shen · Shiqi Shen · Wenwu Zhu -
2023 Poster: Multi-task Graph Neural Architecture Search with Task-aware Collaboration and Curriculum »
Yijian Qin · Xin Wang · Ziwei Zhang · Hong Chen · Wenwu Zhu -
2023 Poster: Spectral Invariant Learning for Dynamic Graphs under Distribution Shifts »
Zeyang Zhang · Xin Wang · Ziwei Zhang · Zhou Qin · Weigao Wen · Hui Xue' · Haoyang Li · Wenwu Zhu -
2023 Poster: Joint Data-Task Generation for Auxiliary Learning »
Hong Chen · Xin Wang · Yuwei Zhou · Yijian Qin · Chaoyu Guan · Wenwu Zhu -
2022 Spotlight: NAS-Bench-Graph: Benchmarking Graph Neural Architecture Search »
Yijian Qin · Ziwei Zhang · Xin Wang · Zeyang Zhang · Wenwu Zhu -
2022 Poster: On the Convergence of Stochastic Multi-Objective Gradient Manipulation and Beyond »
Shiji Zhou · Wenpeng Zhang · Jiyan Jiang · Wenliang Zhong · Jinjie GU · Wenwu Zhu -
2021 Poster: Asynchronous Decentralized Online Learning »
Jiyan Jiang · Wenpeng Zhang · Jinjie GU · Wenwu Zhu -
2021 Poster: Curriculum Disentangled Recommendation with Noisy Multi-feedback »
Hong Chen · Yudong Chen · Xin Wang · Ruobing Xie · Rui Wang · Feng Xia · Wenwu Zhu -
2021 Poster: Disentangled Contrastive Learning on Graphs »
Haoyang Li · Xin Wang · Ziwei Zhang · Zehuan Yuan · Hang Li · Wenwu Zhu -
2021 Poster: Graph Differentiable Architecture Search with Structure Learning »
Yijian Qin · Xin Wang · Zeyang Zhang · Wenwu Zhu -
2021 Poster: Not All Low-Pass Filters are Robust in Graph Convolutional Networks »
Heng Chang · Yu Rong · Tingyang Xu · Yatao Bian · Shiji Zhou · Xin Wang · Junzhou Huang · Wenwu Zhu -
2020 Poster: Implicit Graph Neural Networks »
Fangda Gu · Heng Chang · Wenwu Zhu · Somayeh Sojoudi · Laurent El Ghaoui -
2019 Poster: Semantic Conditioned Dynamic Modulation for Temporal Sentence Grounding in Videos »
Yitian Yuan · Lin Ma · Jingwen Wang · Wei Liu · Wenwu Zhu -
2019 Poster: Learning Disentangled Representations for Recommendation »
Jianxin Ma · Chang Zhou · Peng Cui · Hongxia Yang · Wenwu Zhu -
2018 Poster: Weakly Supervised Dense Event Captioning in Videos »
Xin Wang · Wenbing Huang · Chuang Gan · Jingdong Wang · Wenwu Zhu · Junzhou Huang