Timezone: »
We develop a general approach to distill symbolic representations of a learned deep model by introducing strong inductive biases. We focus on Graph Neural Networks (GNNs). The technique works as follows: we first encourage sparse latent representations when we train a GNN in a supervised setting, then we apply symbolic regression to components of the learned model to extract explicit physical relations. We find the correct known equations, including force laws and Hamiltonians, can be extracted from the neural network. We then apply our method to a non-trivial cosmology example—a detailed dark matter simulation—and discover a new analytic formula which can predict the concentration of dark matter from the mass distribution of nearby cosmic structures. The symbolic expressions extracted from the GNN using our technique also generalized to out-of-distribution-data better than the GNN itself. Our approach offers alternative directions for interpreting neural networks and discovering novel physical principles from the representations they learn.
Author Information
Miles Cranmer (Princeton University)
Miles Cranmer is an Astro PhD candidate trying to accelerate astrophysics with AI. Miles is from Canada and did his undergraduate in Physics at McGill. He is deeply interested in the automation of science, particularly aspects that are not yet tractable with existing machine learning, such as experiment planning, simulation, and theory. He works on symbolic regression, graph neural networks, normalizing flows, and learned simulation. He is hugely interested in symbolic ML, since, as he argues, symbolic models seem to be a surprisingly efficient basis for describing our universe.
Alvaro Sanchez Gonzalez (DeepMind)
Peter Battaglia (DeepMind)
Rui Xu (Princeton University)
Kyle Cranmer (New York University)
Kyle Cranmer is an Associate Professor of Physics at New York University and affiliated with NYU's Center for Data Science. He is an experimental particle physicists working, primarily, on the Large Hadron Collider, based in Geneva, Switzerland. He was awarded the Presidential Early Career Award for Science and Engineering in 2007 and the National Science Foundation's Career Award in 2009. Professor Cranmer developed a framework that enables collaborative statistical modeling, which was used extensively for the discovery of the Higgs boson in July, 2012. His current interests are at the intersection of physics and machine learning and include inference in the context of intractable likelihoods, development of machine learning models imbued with physics knowledge, adversarial training for robustness to systematic uncertainty, the use of generative models in the physical sciences, and integration of reproducible workflows in the inference pipeline.
David Spergel (Flatiron Institute)
Shirley Ho (Flatiron institute/ New York University/ Carnegie Mellon)
Shirley Ho is a group leader and acting director at Flatiron Institute at Simons foundation, a research professor of physics and an affiliated faculty at Center for Data Science at NYU. Ho also holds associate (adjunct) professorship at Carnegie Mellon University and visiting appointment at Princeton University. She was a senior scientist at Berkeley National Lab from 2016-2018 and a Cooper-Siegel Development chair professor at Carnegie Mellon University before that. Ho was a Seaborg and Chamberlain Fellow from 2008-2011 at Berkeley Lab, after receiving her PhD in Astrophysics from Princeton University in 2008 under supervision of David Spergel. Ho graduated summa cum laude with a B.A. in Physics and a B.A. in Computer Science from UC Berkeley. A cited expert in cosmology, machine learning applications in astrophysics and data science,her interests are using deep learning accelerated simulations to understand the Universe, and other astrophysical phenomena. She tries her best to balance her love for the Universe, the machine and life especially during these crazy times.
More from the Same Authors
-
2020 : Learning Mesh-Based Simulation with Graph Networks »
Tobias Pfaff · Meire Fortunato · Alvaro Sanchez Gonzalez · Peter Battaglia -
2021 : Searching for the Weirdest Stars: A Convolutional Autoencoder-Based Pipeline For Detecting Anomalous Periodic Variable Stars »
Ho-Sang Chan · Siu Hei Cheung · Victoria Villar · Shirley Ho -
2021 : Characterizing γ-ray maps of the Galactic Center with neural density estimation »
Siddharth Mishra-Sharma · Kyle Cranmer -
2021 : A Convolutional Autoencoder-Based Pipeline For Anomaly Detection And Classification Of Periodic Variables »
Ho-Sang Chan · Siu Hei Cheung · Shirley Ho -
2021 : Equivariant and Modular DeepSets with Applications in Cluster Cosmology »
Leander Thiele · Miles Cranmer · Shirley Ho · David Spergel -
2021 : The Quantum Trellis: A classical algorithm for sampling the parton shower with interference effects »
Sebastian Macaluso · Kyle Cranmer -
2021 : Searching for the Weirdest Stars: A Convolutional Autoencoder-Based Pipeline For Detecting Anomalous Periodic Variable Stars »
Ho-Sang Chan · Siu Hei Cheung · Victoria Villar · Shirley Ho -
2022 : A Neural Network Subgrid Model of the Early Stages of Planet Formation »
Thomas Pfeil · Miles Cranmer · Shirley Ho · Philip Armitage · Tilman Birnstiel · Hubert Klahr -
2022 : Recovering Galaxy Cluster Convergence from Lensed CMB with Generative Adversarial Networks »
Liam Parker · Dongwon Han · Shirley Ho · Pablo Lemos -
2022 : Astronomical Image Coaddition with Bundle-Adjusting Radiance Fields »
Harlan Hutton · Harshitha Palegar · Shirley Ho · Miles Cranmer · Peter Melchior · Jenna Eubank -
2022 : Emulating cosmological growth functions with B-Splines »
Ngai Pok Kwan · Chirag Modi · Yin Li · Shirley Ho -
2022 : Computing the Bayes-optimal classifier and exact maximum likelihood estimator with a semi-realistic generative model for jet physics »
Kyle Cranmer · Matthew Drnevich · Lauren Greenspan · Sebastian Macaluso · Duccio Pappadopulo -
2022 : Adversarial Noise Injection for Learned Turbulence Simulations »
Jingtong Su · Julia Kempe · Drummond Fielding · Nikolaos Tsilivis · Miles Cranmer · Shirley Ho -
2022 : Learning Integrable Dynamics with Action-Angle Networks »
Ameya Daigavane · Arthur Kosmala · Miles Cranmer · Tess Smidt · Shirley Ho -
2022 : Pre-training via Denoising for Molecular Property Prediction »
Sheheryar Zaidi · Michael Schaarschmidt · James Martens · Hyunjik Kim · Yee Whye Teh · Alvaro Sanchez Gonzalez · Peter Battaglia · Razvan Pascanu · Jonathan Godwin -
2023 Workshop: AI for Science: from Theory to Practice »
Yuanqi Du · Max Welling · Yoshua Bengio · Marinka Zitnik · Carla Gomes · Jure Leskovec · Maria Brbic · Wenhao Gao · Kexin Huang · Ziming Liu · Rocío Mercado · Miles Cranmer · Shengchao Liu · Lijing Wang -
2022 Spotlight: Inverse Design for Fluid-Structure Interactions using Graph Network Simulators »
Kelsey Allen · Tatiana Lopez-Guevara · Kimberly Stachenfeld · Alvaro Sanchez Gonzalez · Peter Battaglia · Jessica Hamrick · Tobias Pfaff -
2022 Spotlight: Lightning Talks 4B-1 »
Alexandra Senderovich · Zhijie Deng · Navid Ansari · Xuefei Ning · Yasmin Salehi · Xiang Huang · Chenyang Wu · Kelsey Allen · Jiaqi Han · Nikita Balagansky · Tatiana Lopez-Guevara · Tianci Li · Zhanhong Ye · Zixuan Zhou · Feng Zhou · Ekaterina Bulatova · Daniil Gavrilov · Wenbing Huang · Dennis Giannacopoulos · Hans-peter Seidel · Anton Obukhov · Kimberly Stachenfeld · Hongsheng Liu · Jun Zhu · Junbo Zhao · Hengbo Ma · Nima Vahidi Ferdowsi · Zongzhang Zhang · Vahid Babaei · Jiachen Li · Alvaro Sanchez Gonzalez · Yang Yu · Shi Ji · Maxim Rakhuba · Tianchen Zhao · Yiping Deng · Peter Battaglia · Josh Tenenbaum · Zidong Wang · Chuang Gan · Changcheng Tang · Jessica Hamrick · Kang Yang · Tobias Pfaff · Yang Li · Shuang Liang · Min Wang · Huazhong Yang · Haotian CHU · Yu Wang · Fan Yu · Bei Hua · Lei Chen · Bin Dong -
2022 Workshop: Machine Learning and the Physical Sciences »
Atilim Gunes Baydin · Adji Bousso Dieng · Emine Kucukbenli · Gilles Louppe · Siddharth Mishra-Sharma · Benjamin Nachman · Brian Nord · Savannah Thais · Anima Anandkumar · Kyle Cranmer · Lenka Zdeborová · Rianne van den Berg -
2022 Poster: Inverse Design for Fluid-Structure Interactions using Graph Network Simulators »
Kelsey Allen · Tatiana Lopez-Guevara · Kimberly Stachenfeld · Alvaro Sanchez Gonzalez · Peter Battaglia · Jessica Hamrick · Tobias Pfaff -
2021 : Kyle Cranmer »
Kyle Cranmer -
2021 : AI X Cosmology »
Shirley Ho -
2021 Workshop: Machine Learning and the Physical Sciences »
Anima Anandkumar · Kyle Cranmer · Mr. Prabhat · Lenka Zdeborová · Atilim Gunes Baydin · Juan Carrasquilla · Emine Kucukbenli · Gilles Louppe · Benjamin Nachman · Brian Nord · Savannah Thais -
2021 : Coding Session 3 »
Shirley Ho · Miles Cranmer -
2021 : The Problem with Deep Learning for Physics (and how to fix it) »
Miles Cranmer · Shirley Ho -
2021 : Accelerating Simulations in Physics with Deep Learning »
Miles Cranmer · Shirley Ho -
2021 : Coding Session »
Shirley Ho · Miles Cranmer -
2021 : Physics-Informed Inductive Biases in Deep Learning »
Miles Cranmer · Shirley Ho -
2021 Tutorial: ML for Physics and Physics for ML »
Shirley Ho · Miles Cranmer -
2021 : The Intersection of ML and Physics »
Shirley Ho · Miles Cranmer -
2020 : Peter Battaglia - Structured models of physics, objects, and scenes »
Peter Battaglia -
2020 Workshop: Interpretable Inductive Biases and Physically Structured Learning »
Michael Lutter · Alexander Terenin · Shirley Ho · Lei Wang -
2020 : Contributed talk - Unsupervised Resource Allocation with Graph Neural Networks »
Miles Cranmer -
2020 : Peter Battaglia »
Peter Battaglia -
2020 Workshop: Machine Learning and the Physical Sciences »
Anima Anandkumar · Kyle Cranmer · Shirley Ho · Mr. Prabhat · Lenka Zdeborová · Atilim Gunes Baydin · Juan Carrasquilla · Adji Bousso Dieng · Karthik Kashinath · Gilles Louppe · Brian Nord · Michela Paganini · Savannah Thais -
2020 Poster: Flows for simultaneous manifold learning and density estimation »
Johann Brehmer · Kyle Cranmer -
2020 Poster: Set2Graph: Learning Graphs From Sets »
Hadar Serviansky · Nimrod Segol · Jonathan Shlomi · Kyle Cranmer · Eilam Gross · Haggai Maron · Yaron Lipman -
2019 : Learning Symbolic Physics with Graph Networks »
Miles Cranmer -
2019 : Hamiltonian Graph Networks with ODE Integrators »
Alvaro Sanchez Gonzalez -
2019 : Morning Coffee Break & Poster Session »
Eric Metodiev · Keming Zhang · Markus Stoye · Randy Churchill · Soumalya Sarkar · Miles Cranmer · Johann Brehmer · Danilo Jimenez Rezende · Peter Harrington · AkshatKumar Nigam · Nils Thuerey · Lukasz Maziarka · Alvaro Sanchez Gonzalez · Atakan Okan · James Ritchie · N. Benjamin Erichson · Harvey Cheng · Peihong Jiang · Seong Ho Pahng · Samson Koelle · Sami Khairy · Adrian Pol · Rushil Anirudh · Jannis Born · Benjamin Sanchez-Lengeling · Brian Timar · Rhys Goodall · Tamás Kriváchy · Lu Lu · Thomas Adler · Nathaniel Trask · Noëlie Cherrier · Tomohiko Konno · Muhammad Kasim · Tobias Golling · Zaccary Alperstein · Andrei Ustyuzhanin · James Stokes · Anna Golubeva · Ian Char · Ksenia Korovina · Youngwoo Cho · Chanchal Chatterjee · Tom Westerhout · Gorka Muñoz-Gil · Juan Zamudio-Fernandez · Jennifer Wei · Brian Lee · Johannes Kofler · Bruce Power · Nikita Kazeev · Andrey Ustyuzhanin · Artem Maevskiy · Pascal Friederich · Arash Tavakoli · Willie Neiswanger · Bohdan Kulchytskyy · sindhu hari · Paul Leu · Paul Atzberger -
2019 : Opening Remarks »
Atilim Gunes Baydin · Juan Carrasquilla · Shirley Ho · Karthik Kashinath · Michela Paganini · Savannah Thais · Anima Anandkumar · Kyle Cranmer · Roger Melko · Mr. Prabhat · Frank Wood -
2019 Workshop: Machine Learning and the Physical Sciences »
Atilim Gunes Baydin · Juan Carrasquilla · Shirley Ho · Karthik Kashinath · Michela Paganini · Savannah Thais · Anima Anandkumar · Kyle Cranmer · Roger Melko · Mr. Prabhat · Frank Wood -
2019 : Closing remarks »
Dan Rosenbaum · Marta Garnelo · Peter Battaglia · Kelsey Allen · Ilker Yildirim -
2019 : Peter Battaglia: Graph Networks for Learning Physics »
Peter Battaglia -
2019 : Opening Remarks »
Dan Rosenbaum · Marta Garnelo · Peter Battaglia · Kelsey Allen · Ilker Yildirim -
2019 Workshop: Perception as generative reasoning: structure, causality, probability »
Dan Rosenbaum · Marta Garnelo · Peter Battaglia · Kelsey Allen · Ilker Yildirim -
2019 Poster: Efficient Probabilistic Inference in the Quest for Physics Beyond the Standard Model »
Atilim Gunes Baydin · Lei Shao · Wahid Bhimji · Lukas Heinrich · Saeid Naderiparizi · Andreas Munk · Jialin Liu · Bradley Gram-Hansen · Gilles Louppe · Lawrence Meadows · Philip Torr · Victor Lee · Kyle Cranmer · Mr. Prabhat · Frank Wood -
2018 : Talk 5: Peter Battaglia - Structure in Physical Intelligence »
Peter Battaglia -
2017 : Object-oriented intelligence »
Peter Battaglia -
2017 : Panel session »
Iain Murray · Max Welling · Juan Carrasquilla · Anatole von Lilienfeld · Gilles Louppe · Kyle Cranmer -
2017 Workshop: Deep Learning for Physical Sciences »
Atilim Gunes Baydin · Mr. Prabhat · Kyle Cranmer · Frank Wood -
2017 Poster: Learning to Pivot with Adversarial Networks »
Gilles Louppe · Michael Kagan · Kyle Cranmer -
2017 Poster: A simple neural network module for relational reasoning »
Adam Santoro · David Raposo · David Barrett · Mateusz Malinowski · Razvan Pascanu · Peter Battaglia · Timothy Lillicrap -
2017 Poster: Imagination-Augmented Agents for Deep Reinforcement Learning »
Sébastien Racanière · Theophane Weber · David Reichert · Lars Buesing · Arthur Guez · Danilo Jimenez Rezende · Adrià Puigdomènech Badia · Oriol Vinyals · Nicolas Heess · Yujia Li · Razvan Pascanu · Peter Battaglia · Demis Hassabis · David Silver · Daan Wierstra -
2017 Spotlight: A simple neural network module for relational reasoning »
Adam Santoro · David Raposo · David Barrett · Mateusz Malinowski · Razvan Pascanu · Peter Battaglia · Timothy Lillicrap -
2017 Oral: Imagination-Augmented Agents for Deep Reinforcement Learning »
Sébastien Racanière · Theophane Weber · David Reichert · Lars Buesing · Arthur Guez · Danilo Jimenez Rezende · Adrià Puigdomènech Badia · Oriol Vinyals · Nicolas Heess · Yujia Li · Razvan Pascanu · Peter Battaglia · Demis Hassabis · David Silver · Daan Wierstra -
2017 Poster: Visual Interaction Networks: Learning a Physics Simulator from Video »
Nicholas Watters · Daniel Zoran · Theophane Weber · Peter Battaglia · Razvan Pascanu · Andrea Tacchetti -
2016 Poster: Unsupervised Learning of 3D Structure from Images »
Danilo Jimenez Rezende · S. M. Ali Eslami · Shakir Mohamed · Peter Battaglia · Max Jaderberg · Nicolas Heess -
2016 Invited Talk: Machine Learning and Likelihood-Free Inference in Particle Physics »
Kyle Cranmer -
2016 Poster: Interaction Networks for Learning about Objects, Relations and Physics »
Peter Battaglia · Razvan Pascanu · Matthew Lai · Danilo Jimenez Rezende · koray kavukcuoglu -
2015 : An alternative to ABC for likelihood-free inference »
Kyle Cranmer