Skip to yearly menu bar Skip to main content


Workshop

Workshop on Advancing Neural Network Training (WANT): Computational Efficiency, Scalability, and Resource Optimization

Julia Gusak ⋅ Jean Kossaifi ⋅ Alena Shilova ⋅ Rocco Sedona ⋅ Cristiana Bentes ⋅ Animashree Anandkumar ⋅ Olivier Beaumont
Project Page

Abstract

Video

Chat is not available.

Schedule

Timezone: America/Los_Angeles
6:50 AM
Opening Remarks
Julia Gusak
8:00 AM
Coffee Break
9:31 AM
Training and inference of large language models using 8-bit floating point
Sergio Perez ⋅ Yan Zhang ⋅ James Briggs ⋅ Charles Blake ⋅ Josh Levy-Kramer ⋅ Paul Balanca ⋅ Carlo Luschi ⋅ Stephen Barlow ⋅ Andrew Fitzgibbon
9:37 AM
MatFormer: Nested Transformer for Elastic Inference
Fnu Devvrit ⋅ Sneha Kudugunta ⋅ Aditya Kusupati ⋅ Tim Dettmers ⋅ Kaifeng Chen ⋅ Inderjit Dhillon ⋅ Yulia Tsvetkov ⋅ Hannaneh Hajishirzi ⋅ Sham Kakade ⋅ Ali Farhadi ⋅ Prateek Jain
9:43 AM
Sparse Backpropagation for MoE Training
Liyuan Liu ⋅ Jianfeng Gao ⋅ Weizhu Chen
9:49 AM
Efficient Parallelization Layouts for Large-Scale Distributed Model Training
Johannes Hagemann ⋅ Samuel Weinbach ⋅ Konstantin Dobler ⋅ Maximilian Schall ⋅ Gerard de Melo
9:55 AM
CoTFormer: More Tokens With Attention Make Up For Less Depth
Amirkeivan Mohtashami ⋅ Matteo Pagliardini ⋅ Martin Jaggi
10:00 AM
Lunch
11:30 AM
Poster Session
1:00 PM
Coffee Break
1:30 PM
Efficient LLM Training and Inference on GPUs
Mohammad Shoeybi ⋅ Bryan Catanzaro
2:00 PM
Panel Discussion
Yang You ⋅ Olatunji Ruwase ⋅ Natalia Vassilieva ⋅ Mohammad Shoeybi ⋅ Ana Gainaru ⋅ Lionel Eyraud-Dubois ⋅ Jean Kossaifi
2:50 PM
Closing Remarks
Jean Kossaifi
3:00 PM
Poster Session
AI4HPC: Library to Train AI Models on HPC Systems using CFD Datasets
Eray Inanc ⋅ Rakesh Sarma ⋅ Marcel Aach ⋅ Rocco Sedona ⋅ Andreas Lintermann
Towards Cheaper Inference in Deep Networks with Lower Bit-Width Accumulators
Yaniv Blumenfeld ⋅ Itay Hubara ⋅ Daniel Soudry
LightSeq: : Sequence Level Parallelism for Distributed Training of Long Context Transformers
Dacheng Li ⋅ Rulin Shao ⋅ Anze Xie ⋅ Eric Xing ⋅ Joseph Gonzalez ⋅ Ion Stoica ⋅ Xuezhe Ma ⋅ Hao Zhang
FlexTrain: A Dynamic Training Framework for Heterogeneous Devices Environments
Mert Unsal ⋅ Ali Maatouk ⋅ Antonio De Domenico ⋅ Nicola Piovesan ⋅ Fadhel Ayed
DYAD: A Descriptive Yet Abjuring Density efficient approximation to linear neural network layers
Sarin Chandy ⋅ Varun Prashant Gangal ⋅ Yi Yang ⋅ Gabriel Maggiotti
Improving Deep Ensembles without Communication
Konstantinos Pitas ⋅ Michael Arbel ⋅ Julyan Arbel
ConcatPlexer : Additional Dim1 Batching for Faster ViTs
Donghoon Han ⋅ Seunghyeon Seo ⋅ Donghyeon Jeon ⋅ Jiho Jang ⋅ Chaerin Kong ⋅ Nojun Kwak
InstaTune: Instantaneous Neural Architecture Search During Fine-Tuning
Sharath Nittur Sridhar ⋅ Souvik Kundu ⋅ Sairam Sundaresan ⋅ Maciej Szankin ⋅ Anthony Sarah
ReLoRA: High-Rank Training Through Low-Rank Updates
Vladislav Lialin ⋅ Sherin Muckatira ⋅ Namrata Shivagunde ⋅ Anna Rumshisky
Sparse Iso-FLOP Transformations for Maximizing Training Efficiency
Vithursan Thangarasa ⋅ Shreyas Saxena ⋅ Abhay Gupta ⋅ Sean Lie
Embarrassingly Simple Dataset Distillation
Yunzhen Feng ⋅ Shanmukha Ramakrishna Vedantam ⋅ Julia Kempe
Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs
Suyu Ge ⋅ Yunan Zhang ⋅ Liyuan Liu ⋅ Minjia Zhang ⋅ Jiawei Han ⋅ Jianfeng Gao
A Quadratic Synchronization Rule for Distributed Deep Learning
Xinran Gu ⋅ Kaifeng Lyu ⋅ Sanjeev Arora ⋅ Jingzhao Zhang ⋅ Longbo Huang
DAREL: Data Reduction with Losses for Training Acceleration of Real and Hypercomplex Neural Networks
Alexander Demidovskij ⋅ Aleksei Trutnev ⋅ Artyom Tugaryov ⋅ Igor Salnikov ⋅ Stanislav Pavlov
Accelerating Deep Learning using Ivy
Guillermo Sanchez-Brizuela ⋅ Ved Patwardhan ⋅ Matthew Barrett ⋅ Paul Anderson ⋅ Mustafa Hani ⋅ Daniel Lenton
LeanFlex-GKP: Advancing Hassle-Free Structured Pruning with Simple Flexible Group Count
Jiamu Zhang ⋅ Shaochen (Henry) Zhong ⋅ Andrew Ye ⋅ Zirui Liu ⋅ Kaixiong Zhou ⋅ Xia Hu ⋅ Shuai Xu ⋅ Vipin Chaudhary
Patch Gradient Descent: Training Neural Networks on Very Large Images
Deepak Gupta ⋅ Gowreesh Mago ⋅ Arnav Chavan ⋅ Dilip K. Prasad ⋅ Rajat Thomas
Batched Low-Rank Adaptation of Foundation Models
Yeming Wen ⋅ Swarat Chaudhuri
Local LoRA: Memory-Efficient Fine-Tuning of Large Language Models
Oscar Key ⋅ Jean Kaddour ⋅ Pasquale Minervini
Early Weight Averaging meets High Learning Rates for LLM Pre-training
Sunny Sanyal ⋅ Atula Neerkaje ⋅ Jean Kaddour ⋅ Abhishek Kumar ⋅ Sujay Sanghavi
Bandit-Driven Batch Selection for Robust Learning under Label Noise
Michal Lisicki ⋅ Graham Taylor ⋅ Mihai Nica
Maestro: Uncovering Low-Rank Structures via Trainable Decomposition
Samuel Horváth ⋅ Stefanos Laskaridis ⋅ Shashank Rajput ⋅ Hongyi Wang
DONUT-hole: DONUT Sparsification by Harnessing Knowledge and Optimizing Learning Efficiency
azhar shaikh ⋅ Michael Cochez ⋅ Denis Diachkov ⋅ Michiel de Rijcke ⋅ Sahar Yousefi
Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning
Mengzhou Xia ⋅ Tianyu Gao ⋅ Zhiyuan Zeng ⋅ Danqi Chen
Training Bayesian Neural Networks with Sparse Subspace Variational Inference
Junbo Li ⋅ Zichen Miao ⋅ Qiang Qiu ⋅ Ruqi Zhang
Task Arithmetic with LoRA for Continual Learning
Rajas Chitale ⋅ Ankit Vaidya ⋅ Aditya Kane ⋅ Archana Ghotkar
Cooperative Learning for Cost-Adaptive Inference
Xingli Fang ⋅ Richard Bradford ⋅ Jung-Eun Kim
Generalisable Agents for Neural Network Optimisation
Kale-ab Tessera ⋅ Callum R. Tilbury ⋅ Sasha Abramowitz ⋅ Ruan John de Kock ⋅ Omayma Mahjoub ⋅ Benjamin Rosman ⋅ Sara Hooker ⋅ Arnu Pretorius