Timezone: »

 
Poster
Deep Bidirectional Language-Knowledge Graph Pretraining
Michihiro Yasunaga · Antoine Bosselut · Hongyu Ren · Xikun Zhang · Christopher D Manning · Percy Liang · Jure Leskovec

Wed Nov 30 02:00 PM -- 04:00 PM (PST) @ Hall J #238
in Poster Session 4 »

Pretraining a language model (LM) on text has been shown to help various downstream NLP tasks. Recent works show that a knowledge graph (KG) can complement text data, offering structured background knowledge that provides a useful scaffold for reasoning. However, these works are not pretrained to learn a deep fusion of the two modalities at scale, limiting the potential to acquire fully joint representations of text and KG. Here we propose DRAGON (Deep Bidirectional Language-Knowledge Graph Pretraining), a self-supervised approach to pretraining a deeply joint language-knowledge foundation model from text and KG at scale. Specifically, our model takes pairs of text segments and relevant KG subgraphs as input and bidirectionally fuses information from both modalities. We pretrain this model by unifying two self-supervised reasoning tasks, masked language modeling and KG link prediction. DRAGON outperforms existing LM and LM+KG models on diverse downstream tasks including question answering across general and biomedical domains, with +5% absolute gain on average. In particular, DRAGON achieves notable performance on complex reasoning about language and knowledge (+10% on questions involving long contexts or multi-step reasoning) and low-resource QA (+8% on OBQA and RiddleSense), and new state-of-the-art results on various BioNLP tasks. Our code and trained models are available at https://github.com/michiyasunaga/dragon.

Keywords: Reasoning · question answering · Self-supervised learning · language model · pretraining · Foundation Model · Knowledge graph · biomedical · commonsense
[ Paper]  [ Poster]  [ OpenReview] 

Author Information

Michihiro Yasunaga (Stanford University)
Antoine Bosselut (Swiss Federal Institute of Technology Lausanne)
Hongyu Ren (Stanford University)
Xikun Zhang (Stanford University)
Christopher D Manning (Stanford University)
Percy Liang (Stanford University)
Jure Leskovec (Stanford University/Pinterest)

More from the Same Authors