Decoding Unseen Concepts from fMRI to Image via Encoder-generated fMRI Training

Reconstructing viewed images from fMRI recordings has been a popular research topic with connections to practical applications. Recent studies focus on settings where decoders are both trained and tested on the same semantic concepts. However, in practice a decoder may need to reconstruct concepts not included in the training set of the target subject, which we formalize as the concept generalization setting. By removing a certain concept from training and evaluating decoders on that concept, we show existing decoders suffer a performance drop in this setting, even when the concept was included for subjects other than the evaluation target. Brain encoders trained on the same data degrade far less in relative terms, which prompted us to synthesize fMRI for images of the target concept and add the resulting pairs to decoder training. In experiments across three decoders, four subjects, and twelve concepts, adding synthetic data recovers a significant portion of the lost performance, showing that encoder-generated synthetic data is a powerful tool for concept generalization that requires no additional fMRI scans.