Timezone: »

From voxels to pixels and back: Self-supervision in natural-image reconstruction from fMRI
Roman Beliy · Guy Gaziv · Assaf Hoogi · Francesca Strappini · Tal Golan · Michal Irani

Wed Dec 11 10:45 AM -- 12:45 PM (PST) @ East Exhibition Hall B + C #153

Reconstructing observed images from fMRI brain recordings is challenging. Unfortunately, acquiring sufficient ''labeled'' pairs of {Image, fMRI} (i.e., images with their corresponding fMRI responses) to span the huge space of natural images is prohibitive for many reasons. We present a novel approach which, in addition to the scarce labeled data (training pairs), allows to train fMRI-to-image reconstruction networks also on "unlabeled" data (i.e., images without fMRI recording, and fMRI recording without images). The proposed model utilizes both an Encoder network (image-to-fMRI) and a Decoder network (fMRI-to-image). Concatenating these two networks back-to-back (Encoder-Decoder & Decoder-Encoder) allows augmenting the training data with both types of unlabeled data. Importantly, it allows training on the unlabeled test-fMRI data. This self-supervision adapts the reconstruction network to the new input test-data, despite its deviation from the statistics of the scarce training data.

Author Information

Roman Beliy (weizmann institute)
Guy Gaziv (Weizmann Institute of Science)

Guy Gaziv a postdoctoral researcher in the DiCarlo Lab at MIT. His PhD research focused on the intersection between machine and human vision, and specifically on decoding visual experience from brain activity. Guy earned his BSc in Electrical and Computer Engineering from The Hebrew University of Jerusalem, and his MSc in Physics and PhD in Computer Science from The Weizmann Institute of Science.

Assaf Hoogi (Weizmann Institute)
Francesca Strappini (Weizmann Institute of Science)
Tal Golan (Columbia University)
Michal Irani (Weizmann Institute of Science)

More from the Same Authors