Timezone: »
Word2Vec (W2V) and Glove are popular word embedding algorithms that perform well on a variety of natural language processing tasks. The algorithms are fast, efficient and their embeddings widely used. Moreover, the W2V algorithm has recently been adopted in the field of graph embedding, where it underpins several leading algorithms. However, despite their ubiquity and the relative simplicity of their common architecture, what the embedding parameters of W2V and Glove learn, and why that is useful in downstream tasks largely remains a mystery. We show that different interactions of PMI vectors encode semantic properties that can be captured in low dimensional word embeddings by suitable projection, theoretically explaining why the embeddings of W2V and Glove work, and, in turn, revealing an interesting mathematical interconnection between the semantic relationships of relatedness, similarity, paraphrase and analogy.
Author Information
Carl Allen (University of Edinburgh)
Ivana Balazevic (University of Edinburgh)
Timothy Hospedales (University of Edinburgh)
More from the Same Authors
-
2021 : A Channel Coding Benchmark for Meta-Learning »
Rui Li · Ondrej Bohdal · Rajesh K Mishra · Hyeji Kim · Da Li · Nicholas Lane · Timothy Hospedales -
2021 : Cutting Down on Prompts and Parameters:Simple Few-Shot Learning with Language Models »
Robert Logan · Ivana Balazevic · Eric Wallace · Fabio Petroni · Sameer Singh · Sebastian Riedel -
2023 : Is Scaling Learned Optimizers Worth It? Evaluating The Value of VeLO's 4000 TPU Months »
Fady Rezk · Antreas Antoniou · Henry Gouk · Timothy Hospedales -
2023 Poster: BayesTune: Bayesian Sparse Deep Model Fine-tuning »
Minyoung Kim · Timothy Hospedales -
2023 Poster: FedL2P: Federated Learning to Personalize »
Royson Lee · Minyoung Kim · Da Li · Xinchi Qiu · Timothy Hospedales · Ferenc Huszar · Nicholas Lane -
2021 : Vision-based system identification and 3D keypoint discovery using dynamics constraints »
Miguel Jaques · Martin Asenov · Michael Burke · Timothy Hospedales -
2021 : Cutting Down on Prompts and Parameters:Simple Few-Shot Learning with Language Models »
Robert Logan · Ivana Balazevic · Eric Wallace · Fabio Petroni · Sameer Singh · Sebastian Riedel -
2021 Poster: EvoGrad: Efficient Gradient-Based Meta-Learning and Hyperparameter Optimization »
Ondrej Bohdal · Yongxin Yang · Timothy Hospedales -
2020 : Q/A for invited talk #3 »
Timothy Hospedales -
2020 : Meta-Learning: Representations and Objectives »
Timothy Hospedales -
2020 Poster: Online Meta-Critic Learning for Off-Policy Actor-Critic Methods »
Wei Zhou · Yiying Li · Yongxin Yang · Huaimin Wang · Timothy Hospedales -
2019 Poster: Multi-relational PoincarĂ© Graph Embeddings »
Ivana Balazevic · Carl Allen · Timothy Hospedales