Skip to yearly menu bar Skip to main content


Workshop

The Fourth Workshop on Efficient Natural Language and Speech Processing (ENLSP-IV): Highlighting New Architectures for Future Foundation Models

Mehdi Rezagholizadeh ⋅ Peyman Passban ⋅ Yu Cheng ⋅ Soheila Samiee ⋅ Yue Dong ⋅ Vahid Partovi Nia ⋅ Qun Liu ⋅ Boxing Chen
Project Page [ OpenReview] 

Abstract

Video

Chat is not available.

Schedule

Timezone: America/Los_Angeles
8:15 AM
Opening Remarks
Mehdi Rezagholizadeh
8:30 AM
Efficiency through Learning from Experience
Bhavana Dalvi Mishra ⋅ Peter Clark
9:30 AM
Morning Break
11:30 AM
Sparsified State-Space Models are Efficient Highway Networks
Woomin Song ⋅ Jihoon Tack ⋅ Sangwoo Mo ⋅ Seunghyuk Oh ⋅ Jinwoo Shin
11:36 AM
Longhorn: State Space Models are Amortized Online Learners
Bo Liu ⋅ Rui Wang ⋅ Lemeng Wu ⋅ Yihao Feng ⋅ Peter Stone ⋅ Qiang Liu
11:42 AM
GEAR: An Efficient Error Reduction Framework for KV Cache Compression in LLM Inference
⋅ Qingru Zhang ⋅ Souvik Kundu ⋅ Geonhwa Jeong ⋅ Zaoxing Liu ⋅ Tushar Krishna ⋅ Tuo Zhao
11:48 AM
An Evolved Universal Transformer Memory
Edoardo Cetin ⋅ Qi Sun ⋅ Tianyu Zhao ⋅ Yujin Tang
11:54 AM
OLMoE: Open Mixture-of-Experts Language Models
Niklas Muennighoff ⋅ Luca Soldaini ⋅ Dirk Groeneveld ⋅ Kyle Lo ⋅ Jacob Morrison ⋅ Sewon Min ⋅ Weijia Shi ⋅ Evan Walsh ⋅ Oyvind Tafjord ⋅ Nathan Lambert ⋅ Yuling Gu ⋅ Shane Arora ⋅ Akshita Bhagia ⋅ Dustin Schwenk ⋅ David Wadden ⋅ Alexander Wettig ⋅ Binyuan Hui ⋅ Tim Dettmers ⋅ Douwe Kiela ⋅ Noah Smith ⋅ Pang Wei Koh ⋅ Amanpreet Singh ⋅ Hannaneh Hajishirzi
12:00 PM
Lunch Break
1:30 PM
RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval
Di Liu ⋅ Meng Chen ⋅ Baotong Lu ⋅ Huiqiang Jiang ⋅ Zhenhua Han ⋅ Qianxi Zhang ⋅ Qi Chen ⋅ Chengruidong Zhang ⋅ Bailu Ding ⋅ Kai Zhang ⋅ Chen Chen ⋅ Fan Yang ⋅ Yuqing Yang ⋅ Lili Qiu
1:36 PM
Post-Training Statistical Calibration for Higher Activation Sparsity
Vui Seng Chua ⋅ Yujie Pan ⋅ Nilesh Jain ⋅ Vui Seng Chua
1:42 PM
Bio-xLSTM: Generative modeling, representation and in-context learning of biological and chemical sequences
Niklas Schmidinger ⋅ Lisa Schneckenreiter ⋅ Philipp Seidl ⋅ Johannes Schimunek ⋅ Pieter-Jan Hoedt ⋅ Johannes Brandstetter ⋅ Andreas Mayr ⋅ Sohvi Luukkonen ⋅ Sepp Hochreiter ⋅ Günter Klambauer
1:48 PM
Inference-Friendly Models With MixAttention
Shashank Rajput ⋅ Ying Sheng ⋅ Sean Owen ⋅ Vitaliy Chiley
1:54 PM
One Initialization to Rule them All: Fine-tuning via Explained Variance Adaptation
Fabian Paischer ⋅ Lukas Hauzenberger ⋅ Thomas Schmied ⋅ Benedikt Alkin ⋅ Marc Deisenroth ⋅ Sepp Hochreiter
3:00 PM
Afternoon Break
3:30 PM
Panel Discussion
Joel Hestness ⋅ Navdeep Jaitly ⋅ Marjan Ghazvininejad ⋅ Katie Derthick ⋅ Yue Dong ⋅ Soheila Samiee
4:20 PM
Best Paper Awards and Closing Remarks
Mehdi Rezagholizadeh
BiRNA-BERT: Adaptive Tokenization for Efficient RNA Language Modeling
Toki Tahmid ⋅ Haz Sameen Shahgir ⋅ Sazan Mahbub ⋅ Yue Dong ⋅ Md. Shamsuzzoha Bayzid
Snakes and Ladders: Accelerating SSM Inference with Speculative Decoding
Yangchao Wu ⋅ Yonatan Dukler ⋅ Matthew Trager ⋅ Alessandro Achille ⋅ Wei Xia ⋅ Stefano Soatto
Rephrasing natural text data with different languages and quality levels for Large Language Model pre-training
Michael Pieler ⋅ Marco Bellagente ⋅ Hannah Teufel ⋅ Duy Phung ⋅ Nathan Cooper ⋅ Jonathan Tow ⋅ Paulo Rocha ⋅ Reshinth Adithyan ⋅ Zaid Alyafeai ⋅ Nikhil Pinnaparaju ⋅ Maksym Zhuravinskyi ⋅ Carlos Riquelme Ruiz
ThinK: Thinner Key Cache by Query-Driven Pruning
Yuhui Xu ⋅ Allan Jie ⋅ Hanze Dong ⋅ Lei Wang ⋅ Xudong LU ⋅ Aojun Zhou ⋅ Amrita Saha ⋅ Caiming Xiong ⋅ Doyen Sahoo
Disentangling Questions from Query Generation for Task-Adaptive Retrieval
Yoonsang Lee ⋅ Minsoo Kim ⋅ seung-won hwang
The N-Grammys: Accelerating Autoregressive Inference with Learning-Free Batched Speculation
Lawrence Stewart ⋅ Matthew Trager ⋅ Sujan Gonugondla ⋅ Stefano Soatto
Different Rates for Different Weights: Decoupled Relative Learning Rate Schedules
Jan Ludziejewski ⋅ Jan Małaśnicki ⋅ Maciej Pióro ⋅ Michał Krutul ⋅ Kamil Ciebiera ⋅ Jakub Krajewski ⋅ Marek Cygan ⋅ Kamil Adamczewski ⋅ Sebastian Jaszczur
Distributed Speculative Inference of Large Language Models is Provably Faster
Nadav Timor ⋅ Jonathan Mamou ⋅ Oren Pereg ⋅ Moshe Berchansky ⋅ Daniel Korat ⋅ Moshe Wasserblat ⋅ Tomer Galanti ⋅ Michal Gordon (Kiwkowitz) ⋅ David Harel
XC-Cache: Cross-Attending to Cached Context for Efficient LLM Inference
Joao Monteiro ⋅ Etienne Marcotte ⋅ Pierre-Andre Noel ⋅ Valentina Zantedeschi ⋅ David Vazquez ⋅ Nicolas Chapados ⋅ Christopher Pal ⋅ Perouz Taslakian
Text Summarization With Graph Attention Networks
Mohammadreza Ardestani ⋅ Yllias Chali
How Redundant Is the Transformer Stack in Speech Representation Models?
Albert Kjøller Jacobsen ⋅ Teresa Scheidt ⋅ Lenka Hýlová ⋅ Lars Kai Hansen
Dynamic Speculation Lookahead Accelerates Speculative Decoding of Large Language Models
Jonathan Mamou ⋅ Oren Pereg ⋅ Daniel Korat ⋅ Moshe Berchansky ⋅ Nadav Timor ⋅ Moshe Wasserblat ⋅ Roy Schwartz
Dense Backpropagation Improves Routing for Sparsely-Gated Mixture-of-Experts
Ashwinee Panda ⋅ Vatsal Baherwani ⋅ Zain Sarwar ⋅ Benjamin Therien ⋅ Sambit Sahu ⋅ Stephen Rawls ⋅ Supriyo Chakraborty ⋅ Tom Goldstein
Accelerating Inference of Retrieval-Augmented Generation via Sparse Context Selection
Yun Zhu ⋅ Jia-Chen Gu ⋅ Caitlin Sikora ⋅ Ho Ko ⋅ Yinxiao Liu ⋅ Chu-Cheng Lin ⋅ Lei Shu ⋅ Liangchen Luo ⋅ Lei Meng ⋅ Bang Liu ⋅ Jindong Chen
VL-Mamba: Exploring State Space Models for Multimodal Learning
Yanyuan Qiao ⋅ Zheng Yu ⋅ Zijia Zhao ⋅ Sihan Chen ⋅ Mingzhen Sun ⋅ Longteng Guo ⋅ Qi Wu ⋅ Jing Liu
S2D: Sorted Speculative Decoding For More Efficient Deployment of Large Language Models
Parsa Kavehzadeh ⋅ Mohammadreza Pourreza ⋅ Mojtaba Valipour ⋅ Tianshu Zhu ⋅ Haoli Bai ⋅ Ali Ghodsi ⋅ Boxing Chen ⋅ Mehdi Rezaghoizadeh
Scaling Smart: Accelerating Large Language Model Pre-Training with Small Model Initialization
Mohammad Samragh ⋅ Iman Mirzadeh ⋅ Keivan Alizadeh-Vahid ⋅ Fartash Faghri ⋅ Minsik Cho ⋅ Moin Nabi ⋅ Devang Naik ⋅ Mehrdad Farajtabar
MisD-MoE: A Multimodal Misinformation Detection Framework with Adaptive Feature Selection
Moyang Liu ⋅ Kaiying Yan ⋅ Yukun Liu ⋅ Ruibo Fu ⋅ zhengqi wen ⋅ Xuefei Liu ⋅ Chenxing Li
Approximations may be all you need: Towards Pre-training LLMs with Low-Rank Decomposition and Optimizers
Namrata Shivagunde ⋅ Mayank Kulkarni ⋅ Giannis Karamanolakis ⋅ Jack FitzGerald ⋅ Yannick Versley ⋅ Saleh Soltan ⋅ Volkan Cevher ⋅ Jianhua Lu ⋅ Anna Rumshisky
Partially Shared Query-Key for Lightweight Language Models
Kai Yang ⋅ Vahid Partovi Nia ⋅ Boxing Chen ⋅ Masoud Asgharian
Zipper: A Multi-Tower Decoder Architecture for Fusing Modalities
Vicky Zayats ⋅ Peter Chen ⋅ Melissa Ferrari ⋅ Dirk Padfield
Improving Multi-candidate Speculative Decoding
XiaoFan Lu ⋅ Yixiao Zeng ⋅ marco levorato ⋅ Feiyang Ma ⋅ ZiXu Yu
Efficient Alignment of Large Language Models via Data Sampling
Amrit Khera ⋅ Rajat Ghosh ⋅ Debojyoti Dutta
Towards Low-bit Communication for Tensor Parallel LLM Inference
Harry Dong ⋅ Tyler Johnson ⋅ Minsik Cho ⋅ Emad Soroush
CSKV: Training-Efficient Channel Shrinking for KV Cache in Long-Context Scenarios
Luning Wang ⋅ Shiyao Li ⋅ Xuefei Ning ⋅ Zhihang Yuan ⋅ Shengen Yan ⋅ Guohao Dai ⋅ Yu Wang
StructMoE : Structured Mixture of Experts Using Low Rank Experts
Zain Sarwar ⋅ Ashwinee Panda ⋅ Benjamin Thérien ⋅ Stephen Rawls ⋅ Anirban Das ⋅ Kartik Balasubramaniam ⋅ Berkcan Kapusuzoglu ⋅ Shixiong Zhang ⋅ Sambit Sahu ⋅ MILIND NAPHADE ⋅ Supriyo Chakraborty
A Systematic Evaluation of Decoding-Free Generative Candidate Selection Methods
Mingyu Derek Ma ⋅ Yanna Ding ⋅ Zijie Huang ⋅ Jianxi Gao ⋅ Yizhou Sun ⋅ Wei Wang
Dynamic layer selection in decoder-only transformers
Theodore Glavas ⋅ Joud Chataoui ⋅ florence regol ⋅ Wassim Jabbour ⋅ Antonios Valkanas ⋅ Mark Coates ⋅ Boris Oreshkin
KD-LoRA: A Hybrid Approach to Efficient Fine-Tuning with LoRA and Knowledge Distillation
Rambod Azimi ⋅ Rishav Rishav ⋅ Marek Teichmann ⋅ Samira Ebrahimi Kahou
Sparse Upcycling: Inference Inefficient Finetuning
Sasha Doubov ⋅ Nikhil Sardana ⋅ Vitaliy Chiley
Accelerating the Low-Rank Decomposed Models
Habib Hajimolahoseini ⋅ Walid Ahmed ⋅ Shuangyue Wen ⋅ Yang Liu
CROSS-JEM: Accurate and Efficient Cross-encoders for Short-text Ranking Tasks
Bhawna Paliwal ⋅ Deepak Saini ⋅ Mudit Dhawan ⋅ Siddarth Asokan ⋅ Nagarajan Natarajan ⋅ Surbhi Aggarwal ⋅ Pankaj Malhotra ⋅ Jian Jiao ⋅ Manik Varma
Speculative Streaming: Fast LLM Inference without Auxiliary Models
Nikhil Bhendawade ⋅ Mahyar Najibi ⋅ Irina Belousova ⋅ Qichen Fu ⋅ Henry Mason ⋅ Mohammad Rastegari
Is 3D Convolution with 5D Tensors Really Necessary for Video Analysis?
Habib Hajimolahoseini ⋅ Walid Ahmed ⋅ Shuangyue Wen ⋅ Yang Liu
Hysteresis Activation Function for Efficient Inference
Moshe Kimhi ⋅ Idan Kashani ⋅ Chaim Baskin ⋅ Avi Mendelson
Approximate Top-k for Increased Parallelism
Oscar Key ⋅ Luka Ribar ⋅ Alberto Cattaneo ⋅ Luke Hudlass-Galley ⋅ Douglas Orr
A Simple and Effective L2 Norm-Based Strategy for KV Cache Compression
Alessio Devoto ⋅ Yu Zhao ⋅ Simone Scardapane ⋅ Pasquale Minervini
RAEE: A Robust Retrieval-Augmented Early Exiting Framework for Efficient Inference
Lianming HUANG ⋅ Shangyu Wu ⋅ Yufei Cui ⋅ Ying Xiong ⋅ Xue (Steve) Liu ⋅ Tei-Wei Kuo ⋅ Nan Guan ⋅ Chun Jason XUE
Speculative Diffusion Decoding for Accelerated Language Generation
Jacob K Christopher ⋅ Brian Bartoldson ⋅ Tal Ben-Nun ⋅ Michael Cardei ⋅ Bhavya Kailkhura ⋅ Nando Fioretto
Duo-LLM: A Framework for Studying Adaptive Computation in Large Language Models
Keivan Alizadeh-Vahid ⋅ Iman Mirzadeh ⋅ Hooman Shahrkokhi ⋅ Dmitry Belenko ⋅ Frank Sun ⋅ Minsik Cho ⋅ Mohammad Hossein Sekhavat ⋅ Moin Nabi ⋅ Mehrdad Farajtabar
Post Training Quantization of Large Language Models with Microscaling Formats
Sayeh Sharify ⋅ Utkarsh Saxena ⋅ Zifei Xu ⋅ Wanzin Yazar ⋅ Ilya Soloveychik ⋅ Xin Wang
Computational Bottlenecks of Training Small-scale Large Language Models
Saleh Ashkboos ⋅ Iman Mirzadeh ⋅ Keivan Alizadeh-Vahid ⋅ Mohammad Hossein Sekhavat ⋅ Moin Nabi ⋅ Mehrdad Farajtabar ⋅ Fartash Faghri
QuAILoRA: Quantization-Aware Initialization for LoRA
Neal G. Lawton ⋅ Aishwarya Padmakumar ⋅ Judith Gaspers ⋅ Jack FitzGerald ⋅ Anoop Kumar ⋅ Greg Ver Steeg ⋅ Aram Galstyan
Mai Ho`omāuna i ka `Ai: Language Models Improve Automatic Speech Recognition in Hawaiian
Kaavya Chaparala ⋅ Guido Zarrella ⋅ Bruce Torres Fischer ⋅ Larry Kimura ⋅ Oiwi Parker Jones
Inducing Elasticity in Foundation Models: Post-Training Techniques for Adaptable Inference
Aashiq Muhamed ⋅ Jiarui Liu ⋅ Mona Diab ⋅ Virginia Smith
Enabling Resource-Efficient On-Device Fine-Tuning of LLMs Using Only Inference Engines
Lei Gao ⋅ Amir Ziashahabi ⋅ Yue Niu ⋅ Salman Avestimehr ⋅ Murali Annavaram
EchoAtt: Attend, Copy, then Adjust\\ for More Efficient Large Language Models
Hossein Rajabzadeh ⋅ Aref Jafari ⋅ Aman Sharma ⋅ Benyamin Jami ⋅ HYOCK JU KWON ⋅ Ali Ghodsi ⋅ Boxing Chen ⋅ Mehdi Rezaghoizadeh
ChemTEB: Chemical Text Embedding Benchmark, an Overview of Embedding Models Performance & Efficiency on a Specific Domain
Ali Shiraee Kasmaee ⋅ Mohammad Khodadad ⋅ Mohammad Arshi Saloot ⋅ Nick Sherck ⋅ Stephen Dokas ⋅ Hamidreza Mahyar ⋅ Soheila Samiee
Scaling laws for post-training quantized large language models
Zifei Xu ⋅ Alexander Lan ⋅ Wanzin Yazar ⋅ Tristan Webb ⋅ Sayeh Sharify ⋅ Xin Wang
Beyond Token Generation: Adaptive Chunk-Distilled Language Modeling
Yanhong Li ⋅ Karen Livescu ⋅ Jiawei Zhou
SharedContextBench: How Lossy are Long-context Methods in KV Cache Reuse
Yucheng LI ⋅ Huiqiang Jiang ⋅ Qianhui Wu ⋅ Xufang Luo ⋅ Surin Ahn ⋅ Chengruidong Zhang ⋅ Amir Abdi ⋅ Dongsheng Li ⋅ Jianfeng Gao ⋅ Yuqing Yang ⋅ Lili Qiu
Dynamic Vocabulary Pruning in Early-Exit LLMs
⋅ Karim Abdel Sadek ⋅ Joan Velja ⋅ Matteo Nulli ⋅ Metod Jazbec
FastDraft: How to Train Your Draft
Ofir Zafrir ⋅ Igor Margulis ⋅ Dorin Shteyman ⋅ Guy Boudoukh
LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategy
Rongzhi Zhang ⋅ Kuan Wang ⋅ Liyuan Liu ⋅ Shuohang Wang ⋅ Hao Cheng ⋅ Chao Zhang ⋅ Yelong Shen
Beyond Parameter Count: Implicit Bias in Soft Mixture of Experts
Youngseog Chung ⋅ Dhruv Malik ⋅ Jeff Schneider ⋅ Yuanzhi Li ⋅ Aarti Singh
SuperPos-Prompt: Enhancing Soft Prompt Tuning of Language Models with Superposition of Multi Token Embeddings
Mohammad Ali Sadraei Javaheri ⋅ Ehsaneddin Asgari ⋅ Alice McHardy ⋅ Hamid Rabiee
Revisiting SMoE Language Models by Evaluating Inefficiencies with Task Specific Expert Pruning
Soumajyoti Sarkar ⋅ Leonard Lausen ⋅ Volkan Cevher ⋅ Thomas Brox ⋅ Sheng Zha ⋅ George Karypis
Flash Inference: Near Linear Time Inference for Long Convolution Sequence Models and Beyond
Costin-Andrei Oncescu ⋅ Sanket Purandare ⋅ Stratos Idreos ⋅ Sham Kakade