Timezone: »
Deep neural networks have seen great success in recent years; however, training a deep model is often challenging as its performance heavily depends on the hyper-parameters used. In addition, finding the optimal hyper-parameter configuration, even with state-of-the-art (SOTA) hyper-parameter optimization (HPO) algorithms, can be time-consuming, requiring multiple training runs over the entire datasetfor different possible sets of hyper-parameters. Our central insight is that using an informative subset of the dataset for model training runs involved in hyper-parameter optimization, allows us to find the optimal hyper-parameter configuration significantly faster. In this work, we propose AUTOMATA, a gradient-based subset selection framework for hyper-parameter tuning. We empirically evaluate the effectiveness of AUTOMATA in hyper-parameter tuning through several experiments on real-world datasets in the text, vision, and tabular domains. Our experiments show that using gradient-based data subsets for hyper-parameter tuning achieves significantly faster turnaround times and speedups of 3×-30× while achieving comparable performance to the hyper-parameters found using the entire dataset.
Author Information
Krishnateja Killamsetty (University of Texas, Dallas)
Guttu Sai Abhishek (Indian Institute of Technology, Bombay)
Hi, this is Abhishek. I completed my graduation from CSE, IIT Bombay, India. Have a good day!
Aakriti Lnu (Indian Institute of Technology Bombay)
Final year Computer Science undergrad at IIT Bombay
Ganesh Ramakrishnan (Indian Institute of Technology Bombay, Indian Institute of Technology Bombay)
Alexandre Evfimievski (International Business Machines)
Lucian Popa (International Business Machines)
Rishabh Iyer (University of Texas, Dallas)
Bio: Prof. Rishabh Iyer is currently an Assistant Professor at the University of Texas, Dallas, where he leads the CARAML Lab. He is also a Visiting Assistant Professor at the Indian Institute of Technology, Bombay. He completed his Ph.D. in 2015 from the University of Washington, Seattle. He is excited in making ML more efficient (both computational and labeling efficiency), robust, and fair. He has received the best paper award at Neural Information Processing Systems (NeurIPS/NIPS) in 2013, the International Conference of Machine Learning (ICML) in 2013, and an Honorable Mention at CODS-COMAD in 2021. He has also won a Microsoft Research Ph.D. Fellowship, a Facebook Ph.D. Fellowship, and the Yang Award for Outstanding Graduate Student from the University of Washington.
More from the Same Authors
-
2021 : A Nested Bi-level Optimization Framework for Robust Few Shot Learning »
Krishnateja Killamsetty · Changbin Li · Chen Zhao · Rishabh Iyer · Feng Chen -
2021 : Targeted Active Learning using Submodular Mutual Information for Imbalanced Medical Image Classification »
Suraj Kothawade · Lakshman Tamil · Rishabh Iyer -
2022 : AutoML for Climate Change: A Call to Action »
Renbo Tu · Nicholas Roberts · Vishak Prasad C · Sibasis Nayak · Paarth Jain · Frederic Sala · Ganesh Ramakrishnan · Ameet Talwalkar · Willie Neiswanger · Colin White -
2022 : Using Informative Data Subsets for Efficient Training of Large Language Models: An Initial Study »
H S V N S Kowndinya Renduchintala · Krishnateja Killamsetty · Sumit Bhatia · Milan Aggarwal · Ganesh Ramakrishnan · Rishabh Iyer -
2022 : TALISMAN: Targeted Active Learning for Object Detection with Rare Classes and Slices using Submodular Mutual Information »
Suraj Kothawade · Saikat Ghosh · Sumit Shekhar · Yu Xiang · Rishabh Iyer -
2023 Poster: Learning to Select a Subset of Training Examples to Generalize Efficient Model Training »
Eeshaan Jain · Tushar Nandy · Gaurav Aggarwal · Ashish Tendulkar · Rishabh Iyer · Abir De -
2023 Poster: When Do Neural Nets Outperform Boosted Trees on Tabular Data? »
Duncan McElfresh · Sujay Khandagale · Jonathan Valverde · Vishak Prasad C · Ganesh Ramakrishnan · Micah Goldblum · Colin White -
2022 Poster: ORIENT: Submodular Mutual Information Measures for Data Subset Selection under Distribution Shift »
Athresh Karanam · Krishnateja Killamsetty · Harsha Kokel · Rishabh Iyer -
2021 Poster: SIMILAR: Submodular Information Measures Based Active Learning In Realistic Scenarios »
Suraj Kothawade · Nathan Beck · Krishnateja Killamsetty · Rishabh Iyer -
2021 Poster: Learning to Select Exogenous Events for Marked Temporal Point Process »
Ping Zhang · Rishabh Iyer · Ashish Tendulkar · Gaurav Aggarwal · Abir De -
2021 Poster: RETRIEVE: Coreset Selection for Efficient and Robust Semi-Supervised Learning »
Krishnateja Killamsetty · Xujiang Zhao · Feng Chen · Rishabh Iyer -
2015 Poster: Submodular Hamming Metrics »
Jennifer Gillenwater · Rishabh K Iyer · Bethany Lusch · Rahul Kidambi · Jeffrey A Bilmes -
2015 Spotlight: Submodular Hamming Metrics »
Jennifer Gillenwater · Rishabh K Iyer · Bethany Lusch · Rahul Kidambi · Jeffrey A Bilmes -
2015 Poster: Mixed Robust/Average Submodular Partitioning: Fast Algorithms, Guarantees, and Applications »
Kai Wei · Rishabh K Iyer · Shengjie Wang · Wenruo Bai · Jeffrey A Bilmes -
2014 Poster: Learning Mixtures of Submodular Functions for Image Collection Summarization »
Sebastian Tschiatschek · Rishabh K Iyer · Haochen Wei · Jeffrey A Bilmes -
2013 Poster: Submodular Optimization with Submodular Cover and Submodular Knapsack Constraints »
Rishabh K Iyer · Jeffrey A Bilmes -
2013 Oral: Submodular Optimization with Submodular Cover and Submodular Knapsack Constraints »
Rishabh K Iyer · Jeffrey A Bilmes -
2013 Poster: Curvature and Optimal Algorithms for Learning and Minimizing Submodular Functions »
Rishabh K Iyer · Stefanie Jegelka · Jeffrey A Bilmes -
2012 Poster: Submodular Bregman Divergences with Applications »
Rishabh K Iyer · Jeffrey A Bilmes