Foveated BagNet: Inherent Interpretability Does Not Exclude Global Context

Deploying deep learning in high-stakes settings demands models that are not only accurate but interpretable. Interpretable-by-design models, exemplified by BagNet, address this by restricting each local class predictor to a small spatial patch, yielding inherently explainable predictions---but at the cost of capturing large-scale image features, resulting in a substantial accuracy penalty. We propose FovBagNet, a foveated extension of BagNet inspired by the mammalian retina. Each local predictor depends on a stack of multiscale patches centered at its location, providing high-resolution detail at the center and progressively coarser context toward the periphery. FovBagNet achieves a top-1 ImageNet accuracy of 0.74, closing most of the 11-point gap between BagNet-33 (0.65) and ResNet-50 (0.76), while retaining the inherent interpretability of BagNet in the form of spatially accurate saliency maps. We also identify a conceptual limitation of the original BagNet saliency maps and empirically assess its impact.