`

Timezone: »

 
Poster
Multi-Task Learning as Multi-Objective Optimization
Ozan Sener · Vladlen Koltun

Tue Dec 04 07:45 AM -- 09:45 AM (PST) @ Room 517 AB #113

In multi-task learning, multiple tasks are solved jointly, sharing inductive bias between them. Multi-task learning is inherently a multi-objective problem because different tasks may conflict, necessitating a trade-off. A common compromise is to optimize a proxy objective that minimizes a weighted linear combination of per-task losses. However, this workaround is only valid when the tasks do not compete, which is rarely the case. In this paper, we explicitly cast multi-task learning as multi-objective optimization, with the overall objective of finding a Pareto optimal solution. To this end, we use algorithms developed in the gradient-based multi-objective optimization literature. These algorithms are not directly applicable to large-scale learning problems since they scale poorly with the dimensionality of the gradients and the number of tasks. We therefore propose an upper bound for the multi-objective loss and show that it can be optimized efficiently. We further prove that optimizing this upper bound yields a Pareto optimal solution under realistic assumptions. We apply our method to a variety of multi-task deep learning problems including digit classification, scene understanding (joint semantic segmentation, instance segmentation, and depth estimation), and multi-label classification. Our method produces higher-performing models than recent multi-task learning formulations or per-task training.

Author Information

Ozan Sener (Intel Labs)
Vladlen Koltun (Intel Labs)

More from the Same Authors

  • 2021 Spotlight: Habitat 2.0: Training Home Assistants to Rearrange their Habitat »
    Andrew Szot · Alexander Clegg · Eric Undersander · Erik Wijmans · Yili Zhao · John Turner · Noah Maestre · Mustafa Mukadam · Devendra Singh Chaplot · Oleksandr Maksymets · Aaron Gokaslan · Vladimír Vondruš · Sameer Dharur · Franziska Meier · Wojciech Galuba · Angel Chang · Zsolt Kira · Vladlen Koltun · Jitendra Malik · Manolis Savva · Dhruv Batra
  • 2021 : Habitat 2.0: Training Home Assistants to Rearrange their Habitat »
    Andrew Szot · Alexander Clegg · Eric Undersander · Erik Wijmans · Yili Zhao · Noah Maestre · Mustafa Mukadam · Oleksandr Maksymets · Aaron Gokaslan · Sameer Dharur · Franziska Meier · Wojciech Galuba · Angel Chang · Zsolt Kira · Vladlen Koltun · Jitendra Malik · Manolis Savva · Dhruv Batra
  • 2021 : Habitat 2.0: Training Home Assistants to Rearrange their Habitat »
    Andrew Szot · Alexander Clegg · Eric Undersander · Erik Wijmans · Yili Zhao · Noah Maestre · Mustafa Mukadam · Oleksandr Maksymets · Aaron Gokaslan · Sameer Dharur · Franziska Meier · Wojciech Galuba · Angel Chang · Zsolt Kira · Vladlen Koltun · Jitendra Malik · Manolis Savva · Dhruv Batra
  • 2021 Poster: Geometry Processing with Neural Fields »
    Guandao Yang · Serge Belongie · Bharath Hariharan · Vladlen Koltun
  • 2021 Poster: Habitat 2.0: Training Home Assistants to Rearrange their Habitat »
    Andrew Szot · Alexander Clegg · Eric Undersander · Erik Wijmans · Yili Zhao · John Turner · Noah Maestre · Mustafa Mukadam · Devendra Singh Chaplot · Oleksandr Maksymets · Aaron Gokaslan · Vladimír Vondruš · Sameer Dharur · Franziska Meier · Wojciech Galuba · Angel Chang · Zsolt Kira · Vladlen Koltun · Jitendra Malik · Manolis Savva · Dhruv Batra
  • 2021 Poster: Differentiable Simulation of Soft Multi-body Systems »
    Yiling Qiao · Junbang Liang · Vladlen Koltun · Ming Lin
  • 2020 Poster: Modeling and Optimization Trade-off in Meta-learning »
    Katelyn Gao · Ozan Sener
  • 2020 Poster: Hausdorff Dimension, Heavy Tails, and Generalization in Neural Networks »
    Umut Simsekli · Ozan Sener · George Deligiannidis · Murat Erdogdu
  • 2020 Spotlight: Hausdorff Dimension, Heavy Tails, and Generalization in Neural Networks »
    Umut Simsekli · Ozan Sener · George Deligiannidis · Murat Erdogdu
  • 2019 : Poster and Coffee Break 2 »
    Karol Hausman · Kefan Dong · Ken Goldberg · Lihong Li · Lin Yang · Lingxiao Wang · Lior Shani · Liwei Wang · Loren Amdahl-Culleton · Lucas Cassano · Marc Dymetman · Marc Bellemare · Marcin Tomczak · Margarita Castro · Marius Kloft · Marius-Constantin Dinu · Markus Holzleitner · Martha White · Mengdi Wang · Michael Jordan · Mihailo Jovanovic · Ming Yu · Minshuo Chen · Moonkyung Ryu · Muhammad Zaheer · Naman Agarwal · Nan Jiang · Niao He · Niko Yasui · Nikos Karampatziakis · Nino Vieillard · Ofir Nachum · Olivier Pietquin · Ozan Sener · Pan Xu · Parameswaran Kamalaruban · Paul Mineiro · Paul Rolland · Philip Amortila · Pierre-Luc Bacon · Prakash Panangaden · Qi Cai · Qiang Liu · Quanquan Gu · Raihan Seraj · Richard Sutton · Rick Valenzano · Robert Dadashi · Rodrigo Toro Icarte · Roshan Shariff · Roy Fox · Ruosong Wang · Saeed Ghadimi · Samuel Sokota · Sean Sinclair · Sepp Hochreiter · Sergey Levine · Sergio Valcarcel Macua · Sham Kakade · Shangtong Zhang · Sheila McIlraith · Shie Mannor · Shimon Whiteson · Shuai Li · Shuang Qiu · Sibon Li · Sid Banerjee · Sitao Luan · Tamer Basar · Thinh Doan · Tianhe (Kevin) Yu · Tianyi Liu · Tom Zahavy · Toryn Klassen · Tuo Zhao · Vicenç Gómez · Vincent Liu · Volkan Cevher · Wesley Suttle · Xiao-Wen Chang · Xiaohan Wei · Xiaotong Liu · Xingguo Li · Xinyi Chen · Xingyou Song · Yao Liu · YiDing Jiang · Yihao Feng · Yilun Du · Yinlam Chow · Yinyu Ye · Yishay Mansour · yixuan.lin.1 · Yonathan Efroni · Yongxin Chen · Yuanhao Wang · Bo Dai · Chen-Yu Wei · Harsh Shrivastava · Hongyang Zhang · Qinqing Zheng · Satpathi SATPATHI · Xueqing Liu · Andreu Vall
  • 2018 Poster: Combinatorial Optimization with Graph Convolutional Networks and Guided Tree Search »
    Zhuwen Li · Qifeng Chen · Vladlen Koltun
  • 2018 Poster: Generalizing to Unseen Domains via Adversarial Data Augmentation »
    Riccardo Volpi · Hong Namkoong · Ozan Sener · John Duchi · Vittorio Murino · Silvio Savarese
  • 2016 Poster: Learning Transferrable Representations for Unsupervised Domain Adaptation »
    Ozan Sener · Hyun Oh Song · Ashutosh Saxena · Silvio Savarese