We introduce a novel method for constructing symmetry-preserving attention networks which reflect the natural invariances of the jet-parton assignment problem to efficiently find assignments without evaluating all permutations. This general approach is applicable to arbitrarily complex configurations and significantly outperforms current methods, improving reconstruction efficiency between 19% - 35% on benchmark problems while decreasing inference time by two to five orders of magnitude, making many important and previously intractable cases tractable.