Fusing simultaneously acquired EEG and fMRI via hierarchical deep transcoding

Functional magnetic resonance imaging (fMRI) is a neuroimaging modality that enables full-brain coverage at relatively high spatial resolution (millimeters), though its temporal resolution is somewhat limited due to the sluggishness of the hemodynamic response . Conversely, electroencephalography (EEG) is a neuroimaging modality with high temporal resolution (milliseconds) and low spatial resolution as it records electrical signals from electrodes on the surface of the scalp. In light of the complementarity between the two modalities, when acquired simultaneously, EEG and fMRI potentially can compensate for the shortcoming in one modality using the merits of the other. Here we propose a model that enables high spatiotemporal resolution recovery of the latent neural source space via transcoding of simultaneous EEG/fMRI data. Specifically a latent source space with millimeter and millisecond resolution is generated through a hierarchical deep transcoding process based on a cyclic Convolutional Neural Network (CNN). An important property of the model is that instead of it being a "black box", it is interpretable and can be seen to extract meaningful features, such as hemodynamic impulse response functions (HRF) from the data.