Timezone: »
Neural Ordinary Differential Equations (NODEs) are a new class of models that transform data continuously through infinite-depth architectures. The continuous nature of NODEs has made them particularly suitable for learning the dynamics of complex physical systems. While previous work has mostly been focused on first order ODEs, the dynamics of many systems, especially in classical physics, are governed by second order laws. In this work, we consider Second Order Neural ODEs (SONODEs). We show how the adjoint sensitivity method can be extended to SONODEs and prove that the optimisation of a first order coupled ODE is equivalent and computationally more efficient. Furthermore, we extend the theoretical understanding of the broader class of Augmented NODEs (ANODEs) by showing they can also learn higher order dynamics with a minimal number of augmented dimensions, but at the cost of interpretability. This indicates that the advantages of ANODEs go beyond the extra space offered by the augmented dimensions, as originally thought. Finally, we compare SONODEs and ANODEs on synthetic and real dynamical systems and demonstrate that the inductive biases of the former generally result in faster training and better performance.
Author Information
Alexander Norcliffe (University College London)
I'm a PhD student in Machine Learning for Medicine. I am co-supervised by Mihaela Van der Schaar and Pietro Lio in the Cambridge Centre for AI in Medicine.
Cristian Bodnar (University of Cambridge)
Ben Day (University of Cambridge)
Nikola Simidjievski (University of Cambridge)
Pietro Lió (University of Cambridge)
More from the Same Authors
-
2021 : Interpretable Data Analysis for Bench-to-Bedside Research »
Zohreh Shams · Botty Dimanov · Nikola Simidjievski · Helena Andres-Terre · Paul Scherer · Urška Matjašec · Mateja Jamnik · Pietro Lió -
2021 : Structure-aware generation of drug-like molecules »
Pavol Drotar · Arian Jamasb · Ben Day · Catalina Cangea · Pietro Lió -
2021 : 3D Pre-training improves GNNs for Molecular Property Prediction »
Hannes Stärk · Dominique Beaini · Gabriele Corso · Prudencio Tossou · Christian Dallago · Stephan Günnemann · Pietro Lió -
2021 : 3D Pre-training improves GNNs for Molecular Property Prediction »
Hannes Stärk · Gabriele Corso · Christian Dallago · Stephan Günnemann · Pietro Lió -
2021 : Approximate Latent Force Model Inference »
Jacob Moss · Felix Opolka · Pietro Lió -
2022 : Learning Feynman Diagrams using Graph Neural Networks »
Alexander Norcliffe · Harrison Mitchell · Pietro Lió -
2022 : A physics-informed search for metric solutions to Ricci flow, their embeddings, and visualisation »
Aarjav Jain · Challenger Mishra · Pietro Lió -
2022 : Improving Classification and Data Imputation for Single-Cell Transcriptomics with Graph Neural Networks »
Han-Bo Li · Ramon Viñas Torné · Pietro Lió -
2022 : Structure-based Drug Design with Equivariant Diffusion Models »
Arne Schneuing · Yuanqi Du · Charles Harris · Arian Jamasb · Ilia Igashov · weitao Du · Tom Blundell · Pietro Lió · Carla Gomes · Max Welling · Michael Bronstein · Bruno Correia -
2022 : A Federated Learning benchmark for Drug-Target Interaction »
Filip Svoboda · Gianluca Mittone · Nicholas Lane · Pietro Lió -
2022 : Benchmarking Graph Neural Network-based Imputation Methods on Single-Cell Transcriptomics Data »
Han-Bo Li · Ramon Viñas Torné · Pietro Lió -
2022 : Sheaf Attention Networks »
Federico Barbero · Cristian Bodnar · Haitz Sáez de Ocáriz Borde · Pietro Lió -
2022 : Surfing on the Neural Sheaf »
Julian Suk · Lorenzo Giusti · Tamir Hemo · Miguel Lopez · Marco La Vecchia · Konstantinos Barmpas · Cristian Bodnar -
2022 : On the Expressive Power of Geometric Graph Neural Networks »
Cristian Bodnar · Chaitanya K. Joshi · Simon Mathis · Taco Cohen · Pietro Liò -
2022 : Human Interventions in Concept Graph Networks »
Lucie Charlotte Magister · Pietro Barbiero · Dmitry Kazhdan · Federico Siciliano · Gabriele Ciravegna · Fabrizio Silvestri · Mateja Jamnik · Pietro Lió -
2023 : PoseCheck: Generative Models for 3D Structure-based Drug Design Produce Unrealistic Poses »
Charles Harris · Kieran Didi · Arian Jamasb · Chaitanya K. Joshi · Simon Mathis · Pietro Lió · Tom Blundell -
2023 : Importance-Guided Diffusion »
Paris Flood · Pietro Lió -
2023 : Modelling biology in novel ways - an AI-first course in Structural Bioinformatics »
Kieran Didi · Charles Harris · Charles Harris · Pietro Lió · Rainer Beck · Rainer Beck -
2023 : Pseudotime Diffusion »
Jacob Moss · Jeremy England · Pietro Lió -
2023 : Beyond Erdos-Renyi: Generalization in Algorithmic Reasoning on Graphs »
Dobrik Georgiev · Pietro Lió · Jakub Bachurski · Junhua Chen · Tunan Shi -
2023 : PoseCheck: Generative Models for 3D Structure-based Drug Design Produce Unrealistic Poses »
Charles Harris · Kieran Didi · Arian Jamasb · Chaitanya Joshi · Simon Mathis · Pietro Lió · Tom Blundell -
2023 : PoseCheck: Generative Models for 3D Structure-based Drug Design Produce Unrealistic Poses »
Charles Harris · Kieran Didi · Arian Jamasb · Chaitanya Joshi · Simon Mathis · Pietro Lió · Tom Blundell -
2023 : A framework for conditional diffusion modelling with applications in protein design »
Kieran Didi · Francisco Vargas · Simon Mathis · Vincent Dutordoir · Emile Mathieu · Urszula Julia Komorowska · Pietro Lió -
2023 : Evaluating Representation Learning on the Protein Structure Universe »
Arian Jamasb · Alex Morehead · Zuobai Zhang · Chaitanya K. Joshi · Kieran Didi · Simon Mathis · Charles Harris · Jian Tang · Jianlin Cheng · Pietro Lió · Tom Blundell -
2023 : A framework for conditional diffusion modelling with applications in motif scaffolding for protein design »
Kieran Didi · Francisco Vargas · Simon Mathis · Vincent Dutordoir · Emile Mathieu · Urszula Julia Komorowska · Pietro Lió -
2023 : PoseCheck: Generative Models for 3D Structure-based Drug Design Produce Unrealistic Poses »
Charles Harris · Kieran Didi · Arian Jamasb · Chaitanya K. Joshi · Simon Mathis · Pietro Lió · Tom Blundell -
2023 : A framework for conditional diffusion modelling with applications in motif scaffolding for protein design »
Kieran Didi · Francisco Vargas · Simon Mathis · Vincent Dutordoir · Emile Mathieu · Urszula Julia Komorowska · Pietro Lió -
2023 : Hybrid Early Fusion for Multi-Modal Biomedical Representations »
Konstantin Hemker · Nikola Simidjievski · Mateja Jamnik -
2023 : SHARCS: Shared Concept Space for\\Explainable Multimodal Learning »
Gabriele Dominici · Pietro Barbiero · Lucie Charlotte Magister · Pietro Lió · Nikola Simidjievski -
2023 : GCondNet: A Novel Method for Improving Neural Networks on Small High-Dimensional Tabular Data »
Andrei Margeloiu · Nikola Simidjievski · Pietro Lió · Mateja Jamnik -
2023 : GCondNet: A Novel Method for Improving Neural Networks on Small High-Dimensional Tabular Data »
Andrei Margeloiu · Nikola Simidjievski · Pietro Lió · Mateja Jamnik -
2023 : Incorporating LLM Priors into Tabular Learners »
Max Zhu · Siniša Stanivuk · Andrija Petrovic · Mladen Nikolic · Pietro Lió -
2023 : Everybody Needs a Little HELP: Explaining Graphs via Hierarchical Concepts »
Jonas Jürß · Lucie Charlotte Magister · Pietro Barbiero · Pietro Lió · Nikola Simidjievski -
2023 : Beyond Erdos-Renyi: Generalization in Algorithmic Reasoning on Graphs »
Dobrik Georgiev · Pietro Lió · Jakub Bachurski · Junhua Chen · Tunan Shi -
2023 : HEALNet – Improving Medical Image Analysis using Multi-Omic Context via Hybrid Early Fusion »
Konstantin Hemker · Nikola Simidjievski · Mateja Jamnik -
2023 : Hybrid Early Fusion for Multi-Modal Biomedical Representations »
Konstantin Hemker · Nikola Simidjievski · Mateja Jamnik -
2023 : SHARCS: Shared Concept Space for\\Explainable Multimodal Learning »
Gabriele Dominici · Pietro Barbiero · Lucie Charlotte Magister · Pietro Lió · Nikola Simidjievski -
2023 Poster: Graph Denoising Diffusion for Inverse Protein Folding »
Kai Yi · Bingxin Zhou · Yiqing Shen · Pietro Lió · Yuguang Wang -
2023 Poster: Interpretable Graph Networks Formulate Universal Algebra Conjectures »
Francesco Giannini · Stefano Fioravanti · Oguzhan Keskin · Alisia Lupidi · Lucie Charlotte Magister · Pietro Lió · Pietro Barbiero -
2023 Poster: Sheaf Hypergraph Networks »
Iulia Duta · Giulia Cassarà · Fabrizio Silvestri · Pietro Lió -
2022 : On the Expressive Power of Geometric Graph Neural Networks »
Cristian Bodnar · Chaitanya K. Joshi · Simon Mathis · Taco Cohen · Pietro Liò -
2022 : Sheaf Attention Networks »
Federico Barbero · Cristian Bodnar · Haitz Sáez de Ocáriz Borde · Pietro Lió -
2022 : Dynamic outcomes-based clustering of disease progression in mechanically ventilated patients »
Emma Rocheteau · Ioana Bica · Pietro Lió · Ari Ercole -
2022 Poster: Concept Embedding Models: Beyond the Accuracy-Explainability Trade-Off »
Mateo Espinosa Zarlenga · Pietro Barbiero · Gabriele Ciravegna · Giuseppe Marra · Francesco Giannini · Michelangelo Diligenti · Zohreh Shams · Frederic Precioso · Stefano Melacci · Adrian Weller · Pietro Lió · Mateja Jamnik -
2022 Poster: Neural Sheaf Diffusion: A Topological Perspective on Heterophily and Oversmoothing in GNNs »
Cristian Bodnar · Francesco Di Giovanni · Benjamin Chamberlain · Pietro Lió · Michael Bronstein -
2022 Poster: Graphein - a Python Library for Geometric Deep Learning and Network Analysis on Biomolecular Structures and Interaction Networks »
Arian Jamasb · Ramon Viñas Torné · Eric Ma · Yuanqi Du · Charles Harris · Kexin Huang · Dominic Hall · Pietro Lió · Tom Blundell -
2022 Poster: Composite Feature Selection Using Deep Ensembles »
Fergus Imrie · Alexander Norcliffe · Pietro Lió · Mihaela van der Schaar -
2022 Poster: SizeShiftReg: a Regularization Method for Improving Size-Generalization in Graph Neural Networks »
Davide Buffelli · Pietro Lió · Fabio Vandin -
2021 : Neural ODE Processes: A Short Summary »
Alexander Norcliffe · Cristian Bodnar · Ben Day · Jacob Moss · Pietro Lió -
2021 : On Second Order Behaviour in Augmented Neural ODEs: A Short Summary »
Alexander Norcliffe · Cristian Bodnar · Ben Day · Nikola Simidjievski · Pietro Lió -
2021 : Structure-aware generation of drug-like molecules »
Pavol Drotar · Arian Jamasb · Ben Day · Catalina Cangea · Pietro Lió -
2021 Poster: Weisfeiler and Lehman Go Cellular: CW Networks »
Cristian Bodnar · Fabrizio Frasca · Nina Otter · Yuguang Wang · Pietro Liò · Guido Montufar · Michael Bronstein -
2020 Poster: Constraining Variational Inference with Geometric Jensen-Shannon Divergence »
Jacob Deasy · Nikola Simidjievski · Pietro Lió -
2019 : Catered Lunch and Poster Viewing (in Workshop Room) »
Gustavo Stolovitzky · Prabhu Pradhan · Pablo Duboue · Zhiwen Tang · Aleksei Natekin · Elizabeth Bondi-Kelly · Xavier Bouthillier · Stephanie Milani · Heimo Müller · Andreas T. Holzinger · Stefan Harrer · Ben Day · Andrey Ustyuzhanin · William Guss · Mahtab Mirmomeni