Timezone: »
Neural network models are known to reinforce hidden data biases, making them unreliable and difficult to interpret. We seek to build models that `know what they do not know' by introducing inductive biases in the function space. We show that periodic activation functions in Bayesian neural networks establish a connection between the prior on the network weights and translation-invariant, stationary Gaussian process priors. Furthermore, we show that this link goes beyond sinusoidal (Fourier) activations by also covering triangular wave and periodic ReLU activation functions. In a series of experiments, we show that periodic activation functions obtain comparable performance for in-domain data and capture sensitivity to perturbed inputs in deep neural networks for out-of-domain detection.
Author Information
Lassi Meronen (Aalto University)
Martin Trapp (Aalto University)
Arno Solin (Aalto University)
More from the Same Authors
-
2022 : Fantasizing with Dual GPs in Bayesian Optimization and Active Learning »
Paul Chang · Prakhar Verma · ST John · Victor Picheny · Henry Moss · Arno Solin -
2022 : Towards Improved Learning in Gaussian Processes: The Best of Two Worlds »
Rui Li · ST John · Arno Solin -
2022 : Mixture of Gaussian Processes with Probabilistic Circuits for Multi-Output Regression »
Mingye Zhu · Zhongjie Yu · Martin Trapp · Arseny Skryagin · Kristian Kersting -
2021 : Sparse Gaussian Processes for Stochastic Differential Equations »
Prakhar Verma · Vincent ADAM · Arno Solin -
2021 Poster: Dual Parameterization of Sparse Variational Gaussian Processes »
Vincent ADAM · Paul Chang · Mohammad Emtiyaz Khan · Arno Solin -
2021 Poster: Spatio-Temporal Variational Gaussian Processes »
Oliver Hamelijnck · William Wilkinson · Niki Loppi · Arno Solin · Theodoros Damoulas -
2021 Poster: Scalable Inference in SDEs by Direct Matching of the Fokker–Planck–Kolmogorov Equation »
Arno Solin · Ella Tamir · Prakhar Verma -
2020 Poster: Stationary Activations for Uncertainty Calibration in Deep Learning »
Lassi Meronen · Christabella Irwanto · Arno Solin -
2020 Poster: Deep Automodulators »
Ari Heljakka · Yuxin Hou · Juho Kannala · Arno Solin -
2018 Poster: Infinite-Horizon Gaussian Processes »
Arno Solin · James Hensman · Richard Turner