Timezone: »

 
Efficient Few-Shot Learning Without Prompts
Oren Pereg · Daniel Korat · Moshe Wasserblat · Lewis Tunstall · Unso Eun Seo Jo · Luke Bates · Nils Reimers

@
in Second Workshop on Efficient Natural Language and Speech Processing (ENLSP-II) »

Recent few-shot learning methods, such as parameter-efficient fine-tuning (PEFT) and pattern exploiting training (PET), have achieved impressive results in label-scarce settings. However, they are difficult to employ since they are highly sensitive to handcrafted prompts, and typically require billion-parameter language models to achieve high accuracy. To address these shortcomings, we propose SetFit (Sentence Transformer Fine-tuning), an efficient and prompt-free framework for few-shot fine-tuning of Sentence Transformers (ST). SetFit works by first fine-tuning a pretrained ST on a small number of labeled text pairs, in a contrastive Siamese manner. The resulting model is then used to generate rich text embeddings, which are used to train a classification head. This simple framework requires no prompts or verbalizers, and achieves high accuracy with orders of magnitude less parameters and runtime than existing techniques. Our experiments show that SetFit achieves results competitive with PEFT and PET techniques, and outperforms them on a variety of classification tasks.

Keywords: ENLSP-Main
[ Poster] 

Author Information

Oren Pereg (Intel Labs)
Daniel Korat (Intel Labs)
Moshe Wasserblat (INTEL)
Lewis Tunstall
Unso Eun Seo Jo (Hugging Face)
Luke Bates (Ubiquitous Knowledge Processing Lab)
Nils Reimers (TU Darmstadt)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors