Timezone: »

 
Spotlight
On the Strong Correlation Between Model Invariance and Generalization
Weijian Deng · Stephen Gould · Liang Zheng

Thu Dec 08 05:00 PM -- 07:00 PM (PST) @
in Lightning Talks 6B-2 »

Generalization and invariance are two essential properties of machine learning models. Generalization captures a model's ability to classify unseen data while invariance measures consistency of model predictions on transformations of the data. Existing research suggests a positive relationship: a model generalizing well should be invariant to certain visual factors. Building on this qualitative implication we make two contributions. First, we introduce effective invariance (EI), a simple and reasonable measure of model invariance which does not rely on image labels. Given predictions on a test image and its transformed version, EI measures how well the predictions agree and with what level of confidence. Second, using invariance scores computed by EI, we perform large-scale quantitative correlation studies between generalization and invariance, focusing on rotation and grayscale transformations. From a model-centric view, we observe generalization and invariance of different models exhibit a strong linear relationship, on both in-distribution and out-of-distribution datasets. From a dataset-centric view, we find a certain model's accuracy and invariance linearly correlated on different test sets. Apart from these major findings, other minor but interesting insights are also discussed.

Keywords: unsupervised model selection · Correlation Study · Model Generalization · Model Invariance
[ Slides] 

Author Information

Weijian Deng (Australian National University)
Weijian Deng

I am a Computer Science PhD student at Australian National University. I focus on out-of-distribution model generalization. In my PhD research, the overall objective is Understanding Model Decision under Dynamic Testing Environments. The purpose is two-fold. The first is to provide an unsupervised way to predicate model accuracy under dynamic test scenarios. The second is to better understand the strengths and limitations of MP models. This research will significantly advance machine perception knowledge in dataset representation, model design and decision understanding

Stephen Gould (ANU)
Liang Zheng (Australian National University)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors