Abstract: We propose \textit{approximately} group-invariant neural networks for quantum many-body physics problems. Those tailored-made architectures are parameter-efficient, scalable, significantly outperform existing symmetry-unaware neural network architectures and are competitive with the state-of-the-art iPEPS methods as we demonstrate on a perturbed toric code toy model on a $10 \times 10$ lattice. This paves way towards studying traditionally challenging quantum spin liquid problems within interpretable neural network architectures.