Timezone: »
Poster
On the Modularity of Hypernetworks
Tomer Galanti · Lior Wolf
In the context of learning to map an input $I$ to a function $h_I:\mathcal{X}\to \mathbb{R}$, two alternative methods are compared: (i) an embedding-based method, which learns a fixed function in which $I$ is encoded as a conditioning signal $e(I)$ and the learned function takes the form $h_I(x) = q(x,e(I))$, and (ii) hypernetworks, in which the weights $\theta_I$ of the function $h_I(x) = g(x;\theta_I)$ are given by a hypernetwork $f$ as $\theta_I=f(I)$.
In this paper, we define the property of modularity as the ability to effectively learn a different function for each input instance $I$. For this purpose, we adopt an expressivity perspective of this property and extend the theory of~\cite{devore} and provide a lower bound on the complexity (number of trainable parameters) of neural networks as function approximators, by eliminating the requirements for the approximation method to be robust. Our results are then used to compare the complexities of $q$ and $g$, showing that under certain conditions and when letting the functions $e$ and $f$ be as large as we wish, $g$ can be smaller than $q$ by orders of magnitude. This sheds light on the modularity of hypernetworks in comparison with the embedding-based method. Besides, we show that for a structured target function, the overall number of trainable parameters in a hypernetwork is smaller by orders of magnitude than the number of trainable parameters of a standard neural network and an embedding method.
Author Information
Tomer Galanti (Tel Aviv University)
Lior Wolf (Facebook AI Research)
Related Events (a corresponding poster, oral, or spotlight)
-
2020 Oral: On the Modularity of Hypernetworks »
Tue. Dec 8th 02:15 -- 02:30 PM Room Orals & Spotlights: Deep Learning
More from the Same Authors
-
2021 Poster: Meta Internal Learning »
Raphael Bensadoun · Shir Gur · Tomer Galanti · Lior Wolf -
2020 Poster: Generating Correct Answers for Progressive Matrices Intelligence Tests »
Niv Pekar · Yaniv Benny · Lior Wolf -
2020 Poster: Hierarchical Patch VAE-GAN: Generating Diverse Videos from a Single Sample »
Shir Gur · Sagie Benaim · Lior Wolf -
2020 Poster: On Infinite-Width Hypernetworks »
Etai Littwin · Tomer Galanti · Lior Wolf · Greg Yang -
2019 Poster: Hyper-Graph-Network Decoders for Block Codes »
Eliya Nachmani · Lior Wolf -
2018 : Poster Session »
Sujay Sanghavi · Vatsal Shah · Yanyao Shen · Tianchen Zhao · Yuandong Tian · Tomer Galanti · Mufan Li · Gilad Cohen · Daniel Rothchild · Aristide Baratin · Devansh Arpit · Evangelos Papalexakis · Michael Perlmutter · Ashok Vardhan Makkuva · Pim de Haan · Yingyan Lin · Wanmo Kang · Cheolhyoung Lee · Hao Shen · Sho Yaida · Dan Roberts · Nadav Cohen · Philippe Casgrain · Dejiao Zhang · Tengyu Ma · Avinash Ravichandran · Julian Emilio Salazar · Bo Li · Davis Liang · Christopher Wong · Glen Bigan Mbeng · Animesh Garg -
2018 Poster: Automatic Program Synthesis of Long Programs with a Learned Garbage Collector »
Amit Zohar · Lior Wolf -
2018 Poster: One-Shot Unsupervised Cross Domain Translation »
Sagie Benaim · Lior Wolf -
2018 Poster: Regularizing by the Variance of the Activations' Sample-Variances »
Etai Littwin · Lior Wolf -
2017 Poster: One-Sided Unsupervised Domain Mapping »
Sagie Benaim · Lior Wolf -
2017 Spotlight: One-Sided Unsupervised Domain Mapping »
Sagie Benaim · Lior Wolf