Timezone: »

 
Oral
Eigen-Distortions of Hierarchical Representations
Alexander Berardino · Valero Laparra · Johannes Ballé · Eero Simoncelli

Tue Dec 05 04:50 PM -- 05:05 PM (PST) @ Hall A
in Deep Learning, Applications »

We develop a method for comparing hierarchical image representations in terms of their ability to explain perceptual sensitivity in humans. Specifically, we utilize Fisher information to establish a model-derived prediction of local sensitivity to perturbations around a given natural image. For a given image, we compute the eigenvectors of the Fisher information matrix with largest and smallest eigenvalues, corresponding to the model-predicted most- and least-noticeable image distortions, respectively. For human subjects, we then measure the amount of each distortion that can be reliably detected when added to the image, and compare these thresholds to the predictions of the corresponding model. We use this method to test the ability of a variety of representations to mimic human perceptual sensitivity. We find that the early layers of VGG16, a deep neural network optimized for object recognition, provide a better match to human perception than later layers, and a better match than a 4-stage convolutional neural network (CNN) trained on a database of human ratings of distorted image quality. On the other hand, we find that simple models of early visual processing, incorporating one or more stages of local gain control, trained on the same database of distortion ratings, predict human sensitivity significantly better than both the CNN and all layers of VGG16.

Author Information

Alexander Berardino (New York University)
Valero Laparra (Universitat de València)
Johannes Ballé (Google)
Eero Simoncelli (HHMI / New York University)

Eero P. Simoncelli received the B.S. degree in Physics in 1984 from Harvard University, studied applied mathematics at Cambridge University for a year and a half, and then received the M.S. degree in 1988 and the Ph.D. degree in 1993, both in Electrical Engineering from the Massachusetts Institute of Technology. He was an Assistant Professor in the Computer and Information Science department at the University of Pennsylvania from 1993 until 1996. He moved to New York University in September of 1996, where he is currently a Professor in Neural Science, Mathematics, and Psychology. In August 2000, he became an Associate Investigator of the Howard Hughes Medical Institute, under their new program in Computational Biology. In Fall 2020, he resigned his HHMI appointment to become the scientific director of the Center for Computational Neuroscience at the Flatiron Institute, of the Simons Foundation. His research interests span a wide range of topics in the representation and analysis of visual images, in both machine and biological systems.

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors