Timezone: »
We propose a novel method for gradient-based optimization of black-box simulators using differentiable local surrogate models. In fields such as physics and engineering, many processes are modeled with non-differentiable simulators with intractable likelihoods. Optimization of these forward models is particularly challenging, especially when the simulator is stochastic. To address such cases, we introduce the use of deep generative models to iteratively approximate the simulator in local neighborhoods of the parameter space. We demonstrate that these local surrogates can be used to approximate the gradient of the simulator, and thus enable gradient-based optimization of simulator parameters. In cases where the dependence of the simulator on the parameter space is constrained to a low dimensional submanifold, we observe that our method attains minima faster than baseline methods, including Bayesian optimization, numerical optimization and approaches using score function gradient estimators.
Author Information
Sergey Shirobokov (Imperial College London)
Vladislav Belavin (National Research University Higher School of Economics)
Michael Kagan (SLAC / Stanford)
Andrei Ustyuzhanin (National Research University Higher School of Economics)
Atilim Gunes Baydin (University of Oxford)
More from the Same Authors
-
2021 : Learning the solar latent space: sigma-variational autoencoders for multiple channel solar imaging »
Edward Brown · Christopher Bridges · Bernard Benson · Atilim Gunes Baydin -
2021 : Simultaneous Multivariate Forecast of Space Weather Indices using Deep Neural Network Ensembles »
Bernard Benson · Christopher Bridges · Atilim Gunes Baydin -
2021 : Dropout and Ensemble Networks for Thermospheric Density Uncertainty Estimation »
Stefano Bonasera · Giacomo Acciarini · Jorge Pérez-Hernández · Bernard Benson · Edward Brown · Eric Sutton · Moriba Jah · Christopher Bridges · Atilim Gunes Baydin -
2022 : Inferring molecular complexity from mass spectrometry data using machine learning »
Timothy Gebhard · Aaron C. Bell · Jian Gong · Jaden J. A. Hastings · George Fricke · Nathalie Cabrol · Scott Sandford · Michael Phillips · Kimberley Warren-Rhodes · Atilim Gunes Baydin -
2023 Workshop: NeurIPS 2023 Workshop: Machine Learning and the Physical Sciences »
Brian Nord · Atilim Gunes Baydin · Adji Bousso Dieng · Emine Kucukbenli · Siddharth Mishra-Sharma · Benjamin Nachman · Kyle Cranmer · Gilles Louppe · Savannah Thais -
2022 Workshop: Machine Learning and the Physical Sciences »
Atilim Gunes Baydin · Adji Bousso Dieng · Emine Kucukbenli · Gilles Louppe · Siddharth Mishra-Sharma · Benjamin Nachman · Brian Nord · Savannah Thais · Anima Anandkumar · Kyle Cranmer · Lenka Zdeborová · Rianne van den Berg -
2021 : Session 3 | Contributed talk: Maximilian Dax, "Amortized Bayesian inference of gravitational waves with normalizing flows" »
Maximilian Dax · Atilim Gunes Baydin -
2021 : Session 3 | Invited talk: Laure Zanna, "The future of climate modeling in the age of machine learning" »
Laure Zanna · Atilim Gunes Baydin -
2021 : Session 3 | Invited talk: Surya Ganguli, "From the geometry of high dimensional energy landscapes to optimal annealing in a dissipative many body quantum optimizer" »
Surya Ganguli · Atilim Gunes Baydin -
2021 : Session 2 | Contributed talk: George Stein, "Self-supervised similarity search for large scientific datasets" »
George Stein · Atilim Gunes Baydin -
2021 : Session 2 | Invited talk: Megan Ansdell, "NASA's efforts & opportunities to support ML in the Physical Sciences" »
Megan Ansdell · Atilim Gunes Baydin -
2021 : Session 1 | Contributed talk: Tian Xie, "Crystal Diffusion Variational Autoencoder for Periodic Material Generation" »
Tian Xie · Atilim Gunes Baydin -
2021 : Session 1 | Invited talk: Bingqing Cheng, "Predicting material properties with the help of machine learning" »
Bingqing Cheng · Atilim Gunes Baydin -
2021 : Session 1 | Invited talk: Max Welling, "Accelerating simulations of nature, both classical and quantum, with equivariant deep learning" »
Max Welling · Atilim Gunes Baydin -
2021 Workshop: Machine Learning and the Physical Sciences »
Anima Anandkumar · Kyle Cranmer · Mr. Prabhat · Lenka Zdeborová · Atilim Gunes Baydin · Juan Carrasquilla · Emine Kucukbenli · Gilles Louppe · Benjamin Nachman · Brian Nord · Savannah Thais -
2021 Poster: Domain Invariant Representation Learning with Domain Density Transformations »
A. Tuan Nguyen · Toan Tran · Yarin Gal · Atilim Gunes Baydin -
2020 : Session 3 | Invited talk: Laura Waller, "Physics-based Learning for Computational Microscopy" »
Laura Waller · Atilim Gunes Baydin -
2020 : Session 2 | Invited talk: Phiala Shanahan, "Generative Flow Models for Gauge Field Theory" »
Phiala Shanahan · Atilim Gunes Baydin -
2020 : Session 2 | Invited talk: Estelle Inack, "Variational Neural Annealing" »
Estelle Inack · Atilim Gunes Baydin -
2020 : Session 1 | Invited talk: Michael Bronstein, "Geometric Deep Learning for Functional Protein Design" »
Michael Bronstein · Atilim Gunes Baydin -
2020 : Session 1 | Invited talk: Lauren Anderson, "3D Milky Way Dust Map using a Scalable Gaussian Process" »
Lauren Anderson · Atilim Gunes Baydin -
2020 Workshop: Machine Learning and the Physical Sciences »
Anima Anandkumar · Kyle Cranmer · Shirley Ho · Mr. Prabhat · Lenka Zdeborová · Atilim Gunes Baydin · Juan Carrasquilla · Adji Bousso Dieng · Karthik Kashinath · Gilles Louppe · Brian Nord · Michela Paganini · Savannah Thais -
2019 : Poster Session »
Pravish Sainath · Mohamed Akrout · Charles Delahunt · Nathan Kutz · Guangyu Robert Yang · Joseph Marino · L F Abbott · Nicolas Vecoven · Damien Ernst · andrew warrington · Michael Kagan · Kyunghyun Cho · Kameron Harris · Leopold Grinberg · John J. Hopfield · Dmitry Krotov · Taliah Muhammad · Erick Cobos · Edgar Walker · Jacob Reimer · Andreas Tolias · Alexander Ecker · Janaki Sheth · Yu Zhang · Maciej Wołczyk · Jacek Tabor · Szymon Maszke · Roman Pogodin · Dane Corneil · Wulfram Gerstner · Baihan Lin · Guillermo Cecchi · Jenna M Reinen · Irina Rish · Guillaume Bellec · Darjan Salaj · Anand Subramoney · Wolfgang Maass · Yueqi Wang · Ari Pakman · Jin Hyung Lee · Liam Paninski · Bryan Tripp · Colin Graber · Alex Schwing · Luke Prince · Gabriel Ocker · Michael Buice · Benjamin Lansdell · Konrad Kording · Jack Lindsey · Terrence Sejnowski · Matthew Farrell · Eric Shea-Brown · Nicolas Farrugia · Victor Nepveu · Jiwoong Im · Kristin Branson · Brian Hu · Ramakrishnan Iyer · Stefan Mihalas · Sneha Aenugu · Hananel Hazan · Sihui Dai · Tan Nguyen · Doris Tsao · Richard Baraniuk · Anima Anandkumar · Hidenori Tanaka · Aran Nayebi · Stephen Baccus · Surya Ganguli · Dean Pospisil · Eilif Muller · Jeffrey S Cheng · Gaël Varoquaux · Kamalaker Dadi · Dimitrios C Gklezakos · Rajesh PN Rao · Anand Louis · Christos Papadimitriou · Santosh Vempala · Naganand Yadati · Daniel Zdeblick · Daniela M Witten · Nicholas Roberts · Vinay Prabhu · Pierre Bellec · Poornima Ramesh · Jakob H Macke · Santiago Cadena · Guillaume Bellec · Franz Scherr · Owen Marschall · Robert Kim · Hannes Rapp · Marcio Fonseca · Oliver Armitage · Jiwoong Im · Thomas Hardcastle · Abhishek Sharma · Wyeth Bair · Adrian Valente · Shane Shang · Merav Stern · Rutuja Patil · Peter Wang · Sruthi Gorantla · Peter Stratton · Tristan Edwards · Jialin Lu · Martin Ester · Yurii Vlasov · Siavash Golkar -
2019 : Morning Coffee Break & Poster Session »
Eric Metodiev · Keming Zhang · Markus Stoye · Randy Churchill · Soumalya Sarkar · Miles Cranmer · Johann Brehmer · Danilo Jimenez Rezende · Peter Harrington · AkshatKumar Nigam · Nils Thuerey · Lukasz Maziarka · Alvaro Sanchez Gonzalez · Atakan Okan · James Ritchie · N. Benjamin Erichson · Harvey Cheng · Peihong Jiang · Seong Ho Pahng · Samson Koelle · Sami Khairy · Adrian Pol · Rushil Anirudh · Jannis Born · Benjamin Sanchez-Lengeling · Brian Timar · Rhys Goodall · Tamás Kriváchy · Lu Lu · Thomas Adler · Nathaniel Trask · Noëlie Cherrier · Tomohiko Konno · Muhammad Kasim · Tobias Golling · Zaccary Alperstein · Andrei Ustyuzhanin · James Stokes · Anna Golubeva · Ian Char · Ksenia Korovina · Youngwoo Cho · Chanchal Chatterjee · Tom Westerhout · Gorka Muñoz-Gil · Juan Zamudio-Fernandez · Jennifer Wei · Brian Lee · Johannes Kofler · Bruce Power · Nikita Kazeev · Andrey Ustyuzhanin · Artem Maevskiy · Pascal Friederich · Arash Tavakoli · Willie Neiswanger · Bohdan Kulchytskyy · sindhu hari · Paul Leu · Paul Atzberger -
2019 : Opening Remarks »
Atilim Gunes Baydin · Juan Carrasquilla · Shirley Ho · Karthik Kashinath · Michela Paganini · Savannah Thais · Anima Anandkumar · Kyle Cranmer · Roger Melko · Mr. Prabhat · Frank Wood -
2019 Workshop: Machine Learning and the Physical Sciences »
Atilim Gunes Baydin · Juan Carrasquilla · Shirley Ho · Karthik Kashinath · Michela Paganini · Savannah Thais · Anima Anandkumar · Kyle Cranmer · Roger Melko · Mr. Prabhat · Frank Wood -
2019 Workshop: Program Transformations for ML »
Pascal Lamblin · Atilim Gunes Baydin · Alexander Wiltschko · Bart van Merriënboer · Emily Fertig · Barak Pearlmutter · David Duvenaud · Laurent Hascoet -
2019 Poster: Efficient Probabilistic Inference in the Quest for Physics Beyond the Standard Model »
Atilim Gunes Baydin · Lei Shao · Wahid Bhimji · Lukas Heinrich · Saeid Naderiparizi · Andreas Munk · Jialin Liu · Bradley Gram-Hansen · Gilles Louppe · Lawrence Meadows · Philip Torr · Victor Lee · Kyle Cranmer · Mr. Prabhat · Frank Wood -
2017 : Panel discussion »
Atilim Gunes Baydin · Adam Paszke · Jonathan Hüser · Jean Utke · Laurent Hascoet · Jeffrey Siskind · Jan Hueckelheim · Andreas Griewank -
2017 : Beyond backprop: automatic differentiation in machine learning »
Atilim Gunes Baydin -
2017 Workshop: Deep Learning for Physical Sciences »
Atilim Gunes Baydin · Mr. Prabhat · Kyle Cranmer · Frank Wood -
2017 Poster: Learning to Pivot with Adversarial Networks »
Gilles Louppe · Michael Kagan · Kyle Cranmer