Growing a Neural Network in Breadth, Depth, and Time

Spatial and temporal resource constraints are critical for both biological and artificial intelligent systems. Here we define differentiable cost terms for breadth, depth, and time within a recurrent convolutional neural network conceived as a finite subset of an infinite lattice architecture. We optimize these costs jointly with task errors via backpropagation, and efficient computational graphs emerge organically through training. We find that breadth, depth, and time can be traded off against each other to achieve a given level of performance. Networks grow in all three dimensions with task complexity and spontaneously take more recurrent steps when inputs are occluded. Surprisingly, time used by the model correlates with human reaction times in an object recognition task. Our framework provides a normative account of how resource constraints shape neural architectures, connecting to questions about brain design in neuroscience, and may help illuminate the diversity of neural solutions found in nature.