Timezone: »

 
Locally Constrained Representations in Reinforcement Learning
Somjit Nath · Samira Ebrahimi Kahou
Event URL: https://openreview.net/forum?id=9YxBdKJx7JT »

The success of Reinforcement Learning (RL) heavily relies on the ability to learn robust representations from the observations of the environment. In most cases, the representations learned purely by the reinforcement learning loss can differ vastly across states depending on how the value functions change. However, the representations learned need not be very specific to the task at hand. Relying only on the RL objective may yield representations that vary greatly across successive time steps. In addition, since the RL loss has a changing target, the representations learned would depend on how good the current values/policies are. Thus, disentangling the representations from the main task would allow them to focus more on capturing transition dynamics which can improve generalization. To this end, we propose locally constrained representations, where an auxiliary loss forces the state representations to be predictable by the representations of the neighbouring states. This encourages the representations to be driven not only by the value/policy learning but also self-supervised learning, which constrains the representations from changing too rapidly. We evaluate the proposed method on several known benchmarks and observe strong performance. Especially in continuous control tasks, our experiments show a significant advantage over a strong baseline.

Author Information

Somjit Nath (Mila/École de technologie supérieure)
Samira Ebrahimi Kahou (McGill University)

More from the Same Authors

  • 2021 : Shift and Scale is Detrimental To Few-Shot Transfer »
    Moslem Yazdanpanah · Christian Desrosiers · Mohammad Havaei · Eugene Belilovsky · Samira Ebrahimi Kahou
  • 2021 : Learning Robust Dynamics through Variational Sparse Gating »
    Arnav Kumar Jain · Shivakanth Sujit · Shruti Joshi · Vincent Michalski · Danijar Hafner · Samira Ebrahimi Kahou
  • 2022 : BERT on a Data Diet: Finding Important Examples by Gradient-Based Pruning »
    Mohsen Fayyaz · Ehsan Aghazadeh · Seyed MohammadAli Modarressi · Mohammad Taher Pilehvar · Yadollah Yaghoobzadeh · Samira Ebrahimi Kahou
  • 2022 : Bridging the Gap Between Offline and Online Reinforcement Learning Evaluation Methodologies »
    Shivakanth Sujit · Pedro Braga · Jörg Bornschein · Samira Ebrahimi Kahou
  • 2022 : Learning from uncertain concepts via test time interventions »
    Ivaxi Sheth · Aamer Abdul Rahman · Laya Rafiee Sevyeri · Mohammad Havaei · Samira Ebrahimi Kahou
  • 2022 : Prioritizing Samples in Reinforcement Learning with Reducible Loss »
    Shivakanth Sujit · Somjit Nath · Pedro Braga · Samira Ebrahimi Kahou
  • 2022 : Pitfalls of conditional computation for multi-modal learning »
    Ivaxi Sheth · Mohammad Havaei · Samira Ebrahimi Kahou
  • 2022 Poster: Learning Robust Dynamics through Variational Sparse Gating »
    Arnav Kumar Jain · Shivakanth Sujit · Shruti Joshi · Vincent Michalski · Danijar Hafner · Samira Ebrahimi Kahou
  • 2021 : From model compression to self-distillation: a review »
    Samira Ebrahimi Kahou
  • 2020 : Spotlight Talk: Ebrahimi Kahou »
    Samira Ebrahimi Kahou
  • 2019 : Lunch Break and Posters »
    Xingyou Song · Elad Hoffer · Wei-Cheng Chang · Jeremy Cohen · Jyoti Islam · Yaniv Blumenfeld · Andreas Madsen · Jonathan Frankle · Sebastian Goldt · Satrajit Chatterjee · Abhishek Panigrahi · Alex Renda · Brian Bartoldson · Israel Birhane · Aristide Baratin · Niladri Chatterji · Roman Novak · Jessica Forde · YiDing Jiang · Yilun Du · Linara Adilova · Michael Kamp · Berry Weinstein · Itay Hubara · Tal Ben-Nun · Torsten Hoefler · Daniel Soudry · Hsiang-Fu Yu · Kai Zhong · Yiming Yang · Inderjit Dhillon · Jaime Carbonell · Yanqing Zhang · Dar Gilboa · Johannes Brandstetter · Alexander R Johansen · Gintare Karolina Dziugaite · Raghav Somani · Ari Morcos · Freddie Kalaitzis · Hanie Sedghi · Lechao Xiao · John Zech · Muqiao Yang · Simran Kaur · Qianli Ma · Yao-Hung Hubert Tsai · Ruslan Salakhutdinov · Sho Yaida · Zachary Lipton · Daniel Roy · Michael Carbin · Florent Krzakala · Lenka Zdeborová · Guy Gur-Ari · Ethan Dyer · Dilip Krishnan · Hossein Mobahi · Samy Bengio · Behnam Neyshabur · Praneeth Netrapalli · Kris Sankaran · Julien Cornebise · Yoshua Bengio · Vincent Michalski · Samira Ebrahimi Kahou · Md Rifat Arefin · Jiri Hron · Jaehoon Lee · Jascha Sohl-Dickstein · Samuel Schoenholz · David Schwab · Dongyu Li · Sang Keun Choe · Henning Petzka · Ashish Verma · Zhichao Lin · Cristian Sminchisescu
  • 2018 : Lunch & Posters »
    Haytham Fayek · German Parisi · Brian Xu · Pramod Kaushik Mudrakarta · Sophie Cerf · Sarah Wassermann · Davit Soselia · Rahaf Aljundi · Mohamed Elhoseiny · Frantzeska Lavda · Kevin J Liang · Arslan Chaudhry · Sanmit Narvekar · Vincenzo Lomonaco · Wesley Chung · Michael Chang · Ying Zhao · Zsolt Kira · Pouya Bashivan · Banafsheh Rafiee · Oleksiy Ostapenko · Andrew Jones · Christos Kaplanis · Sinan Kalkan · Dan Teng · Xu He · Vincent Liu · Somjit Nath · Sungsoo Ahn · Ting Chen · Shenyang Huang · Yash Chandak · Nathan Sprague · Martin Schrimpf · Tony Kendall · Jonathan Richard Schwarz · Michael Li · Yunshu Du · Yen-Chang Hsu · Samira Abnar · Bo Wang
  • 2018 Poster: Towards Deep Conversational Recommendations »
    Raymond Li · Samira Ebrahimi Kahou · Hannes Schulz · Vincent Michalski · Laurent Charlin · Chris Pal