Timezone: »

 
Poster
Modular Networks: Learning to Decompose Neural Computation
Louis Kirsch · Julius Kunze · David Barber

Thu Dec 06 07:45 AM -- 09:45 AM (PST) @ Room 210 #62
in Thu Poster Session A »

Scaling model capacity has been vital in the success of deep learning. For a typical network, necessary compute resources and training time grow dramatically with model size. Conditional computation is a promising way to increase the number of parameters with a relatively small increase in resources. We propose a training algorithm that flexibly chooses neural modules based on the data to be processed. Both the decomposition and modules are learned end-to-end. In contrast to existing approaches, training does not rely on regularization to enforce diversity in module use. We apply modular networks both to image recognition and language modeling tasks, where we achieve superior performance compared to several baselines. Introspection reveals that modules specialize in interpretable contexts.

Keywords: Variational Inference · Recurrent Networks · Supervised Deep Networks · Representation Learning · Latent Variable Models · Optimization for Deep Networks
[ Paper]  [ Poster] 

Author Information

Louis Kirsch (University College London & IDSIA)
Julius Kunze (University College London)
David Barber (University College London)

More from the Same Authors