Welcome statement, introduce the competition -- NeurIPS Large Language Model Efficiency Challenge: 1 LLM + 1GPU + 1Day

Before Kaggle, before the $1m Netflix prize, there was the KDD Cup, the first big annual machine learning competition. Starting in 1997, it brought together each year the world’s best predictive modelers. We now have over 25 years of experience of ML competitions to draw on, with tens of millions of dollars in prizes awarded. I will summarize what we’ve learned from this experience, and explain where, how, and why ML competitions can help advance machine learning research and practice.

Low-rank adaptation (LoRA) stands as one of the most popular and effective methods for efficiently training custom Large Language Models (LLMs). As practitioners of open-source LLMs, we regard LoRA as a crucial technique in our toolkit. In this talk, I will delve into some practical insights gained from running hundreds of experiments with LoRA, addressing questions such as: How much can I save with quantized LoRA? Are Adam optimizers memory-intensive? Should we train for multiple epochs? How do we choose the LoRA rank? Moreover, the talk will include ideas for future experiments and talking points to stimulate discussions in the workshop, such as mechanisms to avoid overfitting in LoRA and strategies for combining LoRA weights from multiple experiments.

In this session, A100 winning team Percent_bdf will reveal the strategies and work that led them to triumph, offering insights and lessons from their experience.

Generative AI is now becoming part and parcel of everyone’s daily life. Large Langage Models such as ChatGPT/GPT4, PaLM, Claude, Llama, Mistral, Falcon and StarCoder are at the core of this owing to their state-of-the-art performance at various Natural Language Processing (NLP) tasks, conversational skills and logical reasoning/coding. The conventional paradigm followed is to pretrain the model on web-scale data followed by finetuning on downstream tasks to get the best performance. The finetuning step becomes infeasible as models get larger due to insufficient access to dedicated hardware thereby preventing widespread availability and usage of these models. Parameter-Efficient Fine-tuning (PEFT) methods enable efficient adaptation of pre-trained language models to various downstream applications without fine-tuning all the model's parameters while maintaining performance. 🤗 PEFT is an open-source project with the vision to democratize access to fine-tuning large AI models on consumer hardware/low-resources while being simple, efficient and adaptable at scale. Here, I will present the development and design considerations that went into building 🤗 PEFT and how it fits in the Generative AI landscape.

In this session, the winning team of 4090 track will reveal the innovative strategies and teamwork that led them to triumph, offering insights and lessons from their experience.

We introduce the large language and multimodal model series Qwen, published and opensourced by Alibaba Group. The Qwen model have achieved competitive performance against both opensource and proprietary LLMs and LMMs in both benchmark and human evaluation. This talk provides a brief overview of the model series and delves into details about building the LLMs, including pretraining, alignment, as well as the opensource. Additionally, it points out the limitations, and discusses the future work for both research community and industry in this field.

Over the past few months, we have released a suite of small language models (SLMs) called “Phi” that achieve unprecedented performance on a variety of benchmarks. Our first model, the 1.3 billion parameter Phi-1, achieved state-of-the-art performance on Python coding among SLMs. We then extended our focus to common sense reasoning and language understanding, and created a new 1.3 billion parameter model named Phi-1.5, with performance comparable to models 5x larger. Our latest model, the 2.7 billion parameter Phi-2, surpasses Phi-1.5 performance on all benchmarks, thanks to new innovations in model scaling and training data curation. In this talk, I will introduce Phi SLMs and discuss two key insights driving their performance: 1) generation and utilization of data with "textbook quality" to elevate the learning process in contrast to conventional web data, and 2) incorporation of best practices for scaling up to enhance overall performance.

Two competing forces are ignored in evaluation: reliability and efficiency. The talk would explain the basics of the open evaluation (HELM) and the analysis done to change an awfully slow evaluation (4K GPU hours for a single model), to hundreds times faster evaluation that you can still rely on its scores. In short, maximize the variability of the data (more datasets prompts, less examples and repetitions) and give more resources to cases you care about (e.g., remove the bottom models after a fast evaluation). The analysis and more details on how to make-check smart evaluation decisions: "efficient benchmarking (of language models)"

https://arxiv.org/abs/2308.11696v3

Please join us to celebrate the community's achievements and the successes of this competition. The event will feature an open floor discussion on the future of the competition. We look forward to celebrating together and exploring what lies ahead.