Timezone: »

 
BERT on a Data Diet: Finding Important Examples by Gradient-Based Pruning
Mohsen Fayyaz · Ehsan Aghazadeh · Seyed MohammadAli Modarressi · Mohammad Taher Pilehvar · Yadollah Yaghoobzadeh · Samira Ebrahimi Kahou

@
in Second Workshop on Efficient Natural Language and Speech Processing (ENLSP-II) »

Current pre-trained language models rely on large datasets for achieving state-of-the-art performance. However, past research has shown that not all examples in a dataset are equally important during training. In fact, it is sometimes possible to prune a considerable fraction of the training set while maintaining the test performance. Established on standard vision benchmarks, two gradient-based scoring metrics for finding important examples are GraNd and its estimated version, EL2N. In this work, we employ these two metrics for the first time in NLP. We demonstrate that these metrics need to be computed after at least one epoch of fine-tuning and they are not reliable in early steps. Furthermore, we show that by pruning a small portion of the examples with the highest GraNd/EL2N scores, we can not only preserve the test accuracy, but also surpass it. This paper details adjustments and implementation choices which enable GraNd and EL2N to be applied to NLP.

Keywords: ENLSP-Main
[ Poster] 

Author Information

Mohsen Fayyaz (University of Tehran)
Ehsan Aghazadeh (University of Tehran)
Seyed MohammadAli Modarressi (IUST)
Mohammad Taher Pilehvar (Tehran Institute for Advanced Studies)
Yadollah Yaghoobzadeh (University of Tehran )
Samira Ebrahimi Kahou (McGill University)

More from the Same Authors