Timezone: »

 
Poster
FourierNets enable the design of highly non-local optical encoders for computational imaging
Diptodip Deb · Zhenfei Jiao · Ruth Sims · Alex Chen · Michael Broxton · Misha B Ahrens · Kaspar Podgorski · Srinivas C Turaga

Thu Dec 01 09:00 AM -- 11:00 AM (PST) @ Hall J #524
in Poster Session 5 »
Differentiable simulations of optical systems can be combined with deep learning-based reconstruction networks to enable high performance computational imaging via end-to-end (E2E) optimization of both the optical encoder and the deep decoder. This has enabled imaging applications such as 3D localization microscopy, depth estimation, and lensless photography via the optimization of local optical encoders. More challenging computational imaging applications, such as 3D snapshot microscopy which compresses 3D volumes into single 2D images, require a highly non-local optical encoder. We show that existing deep network decoders have a locality bias which prevents the optimization of such highly non-local optical encoders. We address this with a decoder based on a shallow neural network architecture using global kernel Fourier convolutional neural networks (FourierNets). We show that FourierNets surpass existing deep network based decoders at reconstructing photographs captured by the highly non-local DiffuserCam optical encoder. Further, we show that FourierNets enable E2E optimization of highly non-local optical encoders for 3D snapshot microscopy. By combining FourierNets with a large-scale multi-GPU differentiable optical simulation, we are able to optimize non-local optical encoders 170$\times$ to 7372$\times$ larger than prior state of the art, and demonstrate the potential for ROI-type specific optical encoding with a programmable microscope.
Keywords: Deep Learning · Computer Vision · Computational Photography · computational microscopy
[ Paper]  [ Poster]  [ OpenReview] 

Author Information

Diptodip Deb (HHMI Janelia Research Campus)
Zhenfei Jiao (Huazhong University of Science and Technology)
Ruth Sims (Institut de la Vision, CNRS)
Alex Chen (HHMI Janelia Research Campus)
Michael Broxton (Google)
Misha B Ahrens (Janelia Farm Research Campus, HHMI)
Kaspar Podgorski (HHMI Janelia Research Campus)
Srinivas C Turaga (HHMI Janelia Research Campus)

More from the Same Authors