The Role of Cortical Varibility in Supporting Few-shot Generalization: Theory and Empirical Evidence

Cortical neurons exhibit a high degree of trial-to-trial variability in response to repeated presentations of the same stimulus. We examine a theory of how such variability can be helpful for generalizing from a small number of examples. We extract three predictions from a simplified Gaussian model of this theory: (1) to minimize generalization error, the optimal neural variability must have a covariance proportional to that of the data points within a class; (2) when considering just two classes, the magnitude of variability must shrink perpendicular to the decision boundary; and (3) the magnitude of variability must shrink in all directions with more examples to generalize from. We then provide evidence from experimental neural data in support of each of these hypotheses. We observe, in the visual cortex of mice, that variability is aligned with in-class variance; that the magnitude of variability shrinks in a task-specific direction with task engagement; and that the magnitude of variability shrinks in all directions with increased stimulus familiarity. Finally, we demonstrate that injecting noise with the appropriate correlation structure into the intermediate layers of a convolutional neural network can promote generalization over rotations of the input. Taken together, the data and simulations provide evidence consistent with the theory that cortical variability supports few-shot generalization.