BrainField: Aligning Neural Representations via Learned Coordinate Charts for Multi-Subject fMRI

Anatomical registration does not align fine-grained functional topography across brains, motivating multi-subject fMRI models to use dense subject-specific projections or post-hoc alignment. We propose BrainField, a bidirectional encoding--decoding framework that represents each subject only by a learned coordinate chart over voxels, initialized from anatomy. Chart-conditioned cross-attention maps irregular voxel sets to a shared grid, while all encoder, decoder, and flow weights are shared. We test whether residual subject variation remains in latent space using a hierarchy of invertible transformations, subject permutations, and variance partitioning. On four Natural Scenes Dataset subjects, BrainField uses 24--32\% of the parameters of four single-subject models while retaining 99\% of decoding performance and reaching up to 111\% of encoding performance. Its latents are natively aligned across subjects, outperforming post-hoc Procrustes- or ridge-aligned single-subject representations, and support direct brain-to-brain translation. Subject identity explains under 0.1\% of latent variance, while the only transformation that improves retrieval carries no detectable subject-specific information. A new subject can be added by fitting only its coordinate chart with the generative flow frozen, retaining 93\% of decoding performance.