Spatiotemporal Information Flows

We develop a novel flow extraction framework, \textit{spatiotemporal information flow} to capture, in an unbiased manner, salient causal relationships between pixels over space and time. Real spatiotemporal dynamical systems such as cellular morphodynamics are complex, nonlinear and evolve over time in response to feedbacks. This makes it highly challenging to model, simulate or fit phenomena from first principle physics. More critically, we often do not know \textit{a priori} and desire to discover the salient variables to include and the key relationships to model. As such causal measures to identify relationships direct from observational multivariate timeseries have been developed. These measures however have largely only been studied in 1D. Here, spatiotemporal information flows present a general, multiscale method to extend 1D causal measures to 2D + time video. Applying spatiotemporal information flows we discover the salient pixel-to-pixel information transfer highways in videos of diverse phenomena from traffic and crowd flow, to collision physics, to fish swarming, to moving camouflaged animals, to human action, embryo development, cell division and cell migration.