Timezone: »

 
Fast Implicit Constrained Optimization of Non-decomposable Objectives for Deep Networks
Yatong Chen · Abhishek Kumar · Yang Liu · Ehsan Amid

@
in Has it Trained Yet? A Workshop for Algorithmic Efficiency in Practical Neural Network Training »
Event URL: https://openreview.net/forum?id=u8kFmPafOc »

We consider a popular family of constrained optimization problems in machine learning that involve optimizing a non-decomposable objective while constraining another. Different from the previous approach that expresses the classifier thresholds as a function of all model parameters, we consider an alternative strategy where the thresholds are expressed as a function of only a subset of the model parameters, i.e., the last layer of the neural network. We propose new training procedures that optimize for the bottom and last layers separately, and solve them using standard gradient based methods. Experiments on a benchmark dataset demonstrate our proposed method achieves performance comparable to the existing approach while being computationally efficient.

Author Information

Yatong Chen (UC Santa Cruz, Google Brain)
Abhishek Kumar (Google Brain)
Yang Liu (UC Santa Cruz)
Ehsan Amid (Google)

More from the Same Authors