In the context of learning continuous-time system dynamics, models incorporating the Hamiltonian inductive bias have been shown to generalise better when modelling conservative data. However, current methods rely on either prior knowledge or trial and error to identify whether imposing such a structure is appropriate. In this work, we demonstrate the effectiveness of Bayesian model selection to do so automatically at training time. We develop a Gaussian process-based model that parameterises the range of structural properties, such as energy conservation or dissipation, that a physical system may exhibit. We train this model with a variational inference scheme and show that the inductive biases inferred from data align with the true underlying properties of the system, thereby providing evidence that the marginal likelihood constitutes a sensible objective for automatic inductive bias selection in dynamical systems.