Timezone: »

 
Poster
On the Downstream Performance of Compressed Word Embeddings
Avner May · Jian Zhang · Tri Dao · Christopher Ré

Thu Dec 12 05:00 PM -- 07:00 PM (PST) @ East Exhibition Hall B + C #185
in Deep Learning -- Embedding Approaches »

Compressing word embeddings is important for deploying NLP models in memory-constrained settings. However, understanding what makes compressed embeddings perform well on downstream tasks is challenging---existing measures of compression quality often fail to distinguish between embeddings that perform well and those that do not. We thus propose the eigenspace overlap score as a new measure. We relate the eigenspace overlap score to downstream performance by developing generalization bounds for the compressed embeddings in terms of this score, in the context of linear and logistic regression. We then show that we can lower bound the eigenspace overlap score for a simple uniform quantization compression method, helping to explain the strong empirical performance of this method. Finally, we show that by using the eigenspace overlap score as a selection criterion between embeddings drawn from a representative set we compressed, we can efficiently identify the better performing embedding with up to 2x lower selection error rates than the next best measure of compression quality, and avoid the cost of training a separate model for each task of interest.

Keywords: Embedding Approaches · Deep Learning · Algorithms -> Classification; Algorithms -> Kernel Methods; Algorithms -> Regression; Applications · Hardware and Systems; App

Author Information

Avner May (Stanford University)
Jian Zhang (Stanford University)
Tri Dao (Stanford University)
Christopher Ré (Stanford)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors