Timezone: »

Differentiable computer vision, graphics, and physics in machine learning
Krishna Murthy Jatavallabhula · Kelsey Allen · Victoria Dean · Johanna Hansen · Shuran Song · Florian Shkurti · Liam Paull · Derek Nowrouzezahrai · Josh Tenenbaum

Fri Dec 11 06:45 AM -- 02:30 PM (PST) @ None
Event URL: https://montrealrobotics.ca/diffcvgp/ »

“Differentiable programs” are parameterized programs that allow themselves to be rewritten by gradient-based optimization. They are ubiquitous in modern-day machine learning. Recently, explicitly encoding our knowledge of the rules of the world in the form of differentiable programs has become more popular. In particular, differentiable realizations of well-studied processes such as physics, rendering, projective geometry, optimization to name a few, have enabled the design of several novel learning techniques. For example, many approaches have been proposed for unsupervised learning of depth estimation from unlabeled videos. Differentiable 3D reconstruction pipelines have demonstrated the potential for task-driven representation learning. A number of differentiable rendering approaches have been shown to enable single-view 3D reconstruction and other inverse graphics tasks (without requiring any form of 3D supervision). Differentiable physics simulators are being built to perform physical parameter estimation from video or for model-predictive control. While these advances have largely occurred in isolation, recent efforts have attempted to bridge the gap between the aforementioned areas. Narrowing the gaps between these otherwise isolated disciplines holds tremendous potential to yield new research directions and solve long-standing problems, particularly in understanding and reasoning about the 3D world.

Hence, we propose the “first workshop on differentiable computer vision, graphics, and physics in machine learning” with the aim of:
1. Narrowing the gap and fostering synergies between the computer vision, graphics, physics, and machine learning communities
2. Debating the promise and perils of differentiable methods, and identifying challenges that need to be overcome
3. Raising awareness about these techniques to the larger ML community
4. Discussing the broader impact of such techniques, and any ethical implications thereof.

Fri 6:45 a.m. - 7:00 a.m.
Opening remarks (Introductory remarks)
Krishna Jatavallabhula, Kelsey Allen, Johanna Hansen, Victoria Dean
Fri 7:00 a.m. - 7:30 a.m.
Sanja Fidler (Invited talk)
Sanja Fidler
Fri 7:30 a.m. - 8:00 a.m.
Andrea Tagliasacchi (Invited talk)
Andrea Tagliasacchi
Fri 8:02 a.m. - 8:31 a.m.
Peter Battaglia (Invited talk)   
Peter Battaglia
Fri 8:32 a.m. - 8:37 a.m.
Peter Battaglia - Q&A (Q&A)
Fri 8:38 a.m. - 8:53 a.m.
Camillo Jose Taylor (Invited talk)   
CJ Taylor
Fri 8:54 a.m. - 8:59 a.m.
Camillo Jose Taylor - Q&A (Q&A)
Fri 9:00 a.m. - 9:13 a.m.
Oral 01: phiflow - A differentiable PDE solving framework for deep learning via physical simulations (Contributed Talk)   
Nils Thuerey
Fri 9:13 a.m. - 9:23 a.m.
Oral 02: Differentiable HDR image synthesis using multi-exposure images (Contributed Talk)   
Jung Hee Kim
Fri 9:23 a.m. - 9:35 a.m.
Oral 03: DELUCA - Differentiable control library - environments, methods, and benchmarking (Contributed Talk)   
Paula Gradu
Fri 9:35 a.m. - 9:44 a.m.
Oral 04: Blendshape-augmented facial action units detection (Contributed Talk)   
Zijun Cui
Fri 9:44 a.m. - 9:57 a.m.
Oral 05: Inverse articulated-body dynamics from video via variational sequential Monte-Carlo (Contributed Talk)   
Dan Biderman
Fri 9:58 a.m. - 10:08 a.m.
Contributed Talk - Q&A (Q&A)
Fri 10:10 a.m. - 10:35 a.m.
Bethany Lusch (Invited talk)   
Bethany Lusch
Fri 10:36 a.m. - 10:41 a.m.
Bethany Lusch - Q&A (Q&A)
Fri 10:42 a.m. - 11:13 a.m.
Yuanming Hu (Invited talk)   
Yuanming Hu
Fri 11:14 a.m. - 11:19 a.m.
Yuanming Hu - Q&A (Q&A)
Fri 11:20 a.m. - 11:39 a.m.
Georgia Gkioxari (Invited talk)   
Georgia Gkioxari
Fri 11:40 a.m. - 11:45 a.m.
Georgia Gkioxari - Q&A (Q&A)
Fri 11:46 a.m. - 12:16 p.m.
Ming Lin (Invited talk)
Ming Lin
Fri 12:16 p.m. - 1:15 p.m.
Panel Discussion
Fri 1:15 p.m. - 2:30 p.m.
Poster session (gather.town) (Poster Session)  link »
Poster 01: Using differentiable physics for self-supervised assimilation of chaotic dynamical systems (Contributed poster)   
Michael McCabe
Poster 02: Learned equivariant rendering without transformation supervision (Contributed poster)   
Cinjon Resnick
Poster 03: Differentiable data augmentation with Kornia (Contributed poster)   
Jian Shi
Poster 04: Semantic adversarial robustness with differentiable ray-tracing (Contributed poster)   
Rahul Venkatesh
Poster 05: Inverse graphics GAN (Contributed poster)   
Sebastian Lunz
Poster 06: Instance-wise depth and motion learning from monocular videos (Contributed poster)   
Seokju Lee
Poster 07: System level differentiable simulation of radio access networks (Contributed poster)   
Dmitriy Rivkin
Poster 08: Solving physics puzzles by reasoning about paths (Contributed poster)   
Augustin Harter
Poster 09: Sparse-input neural network augmentations for differentiable simulators (Contributed poster)   
Eric Heiden, David Millard
Poster 10: Tractable loss function and color image generation of multinary restricted Boltzmann machine (Contributed poster)   
Juno Hwang
Poster 11: Differentiable path tracing by regularizing discontinuities (Contributed poster)   
Peter Quinn
Poster 12: Spring-Rod system identification via differentiable physics engine (Contributed poster)   
Kun Wang
Poster 13: End-to-end differentiable 6DoF object pose estimation with local and global constraints (Contributed poster)   
Anshul Gupta, Joydeep Medhi, Aratrik Chattopadhyay, Vikram Gupta
Poster 14: MSR-Net: Multi-scale relighting network for one-to-one relighting (Contributed poster)   
Nisarg Shah
Poster 15: Towards end-to-end training of proposal-based 3D human pose estimation (Contributed poster)   
Sirdaniel Ajisafe

Author Information

Krishna Jatavallabhula (Mila, Universite de Montreal)
Kelsey Allen (MIT)
Victoria Dean (CMU)
Johanna Hansen (McGill University)
Shuran Song (Columbia University)
Florian Shkurti (University of Toronto)
Liam Paull (Université de Montréal)
Derek Nowrouzezahrai (McGill University)
Josh Tenenbaum (MIT)

Josh Tenenbaum is an Associate Professor of Computational Cognitive Science at MIT in the Department of Brain and Cognitive Sciences and the Computer Science and Artificial Intelligence Laboratory (CSAIL). He received his PhD from MIT in 1999, and was an Assistant Professor at Stanford University from 1999 to 2002. He studies learning and inference in humans and machines, with the twin goals of understanding human intelligence in computational terms and bringing computers closer to human capacities. He focuses on problems of inductive generalization from limited data -- learning concepts and word meanings, inferring causal relations or goals -- and learning abstract knowledge that supports these inductive leaps in the form of probabilistic generative models or 'intuitive theories'. He has also developed several novel machine learning methods inspired by human learning and perception, most notably Isomap, an approach to unsupervised learning of nonlinear manifolds in high-dimensional data. He has been Associate Editor for the journal Cognitive Science, has been active on program committees for the CogSci and NIPS conferences, and has co-organized a number of workshops, tutorials and summer schools in human and machine learning. Several of his papers have received outstanding paper awards or best student paper awards at the IEEE Computer Vision and Pattern Recognition (CVPR), NIPS, and Cognitive Science conferences. He is the recipient of the New Investigator Award from the Society for Mathematical Psychology (2005), the Early Investigator Award from the Society of Experimental Psychologists (2007), and the Distinguished Scientific Award for Early Career Contribution to Psychology (in the area of cognition and human learning) from the American Psychological Association (2008).

More from the Same Authors