Timezone: »

 
Analysis of the Attention in Tabular Language Models
Aneta Koleva · Martin Ringsquandl · Volker Tresp

@
in Table Representation Learning »
Event URL: https://openreview.net/forum?id=-kZV-BVwISM »

Recent transformer-based models for learning table representation have reported state-of-the-art results for different tasks such as table understanding, question answering and semantic parsing. The various proposed models use different architectures, specifically different attention mechanisms. In this paper, we analyze and compare the attention mechanisms used by two different tabular language models. By visualizing the attention maps of the models, we shed a light on the different patterns that the models exhibit. With our analysis on the aggregate attention over two tabular datasets, we provide insights which might help towards building more efficient models tailored for table representation learning.

Keywords: Attention · tabular language models

Author Information

Aneta Koleva (Ludwig Maximilian University of Munich, Siemens AG)
Martin Ringsquandl (Siemens Corporate Research)
Volker Tresp (Siemens AG)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors