Timezone: »
Humans can perform unseen tasks by recalling relevant skills acquired previously and then generalizing them to the target tasks, even if there is no supervision at all. In this paper, we aim to improve this kind of cross-task generalization ability of massive multi-task language models, such as T0 and FLAN, in an unsupervised setting. We propose a retrieval-augmentation method named ReCross that takes a few unlabelled examples as queries to retrieve a small subset of upstream data and uses them to update the multi-task model for better generalization. ReCross is a straightforward yet effective retrieval method that combines both efficient dense retrieval and effective pair-wise reranking. Our results and analysis show that it significantly outperforms both non-retrieval methods and other baseline methods.
Author Information
Bill Yuchen Lin (University of Southern California)
Kangmin Tan (University of Southern California)
Chris Miller
Beiwen Tian (Tsinghua University, Tsinghua University)
Xiang Ren (University of Southern California)
More from the Same Authors
-
2020 : Poster #2 »
Xiang Ren -
2021 Spotlight: Refining Language Models with Compositional Explanations »
Huihan Yao · Ying Chen · Qinyuan Ye · Xisen Jin · Xiang Ren -
2022 Poster: TOIST: Task Oriented Instance Segmentation Transformer with Noun-Pronoun Distillation »
Pengfei Li · Beiwen Tian · Yongliang Shi · Xiaoxue Chen · Hao Zhao · Guyue Zhou · Ya-Qin Zhang -
2022 : PINTO: Faithful Language Reasoning Using Prompt-Generated Rationales »
Peifeng Wang · Aaron Chan · Filip Ilievski · Muhao Chen · Xiang Ren -
2022 : Adaptive Pre-training of Language Models for Better Logical Reasoning »
Soumya Sanyal · Yichong Xu · Shuohang Wang · Ziyi Yang · Reid Pryzant · Wenhao Yu · Chenguang Zhu · Xiang Ren -
2022 : SPRINT: Scalable Semantic Policy Pre-training via Language Instruction Relabeling »
Jesse Zhang · Karl Pertsch · Jiahui Zhang · Taewook Nam · Sung Ju Hwang · Xiang Ren · Joseph Lim -
2022 : Information-Theoretic Evaluation of Free-Text Rationales with Conditional $\mathcal{V}$-Information »
Hanjie Chen · Faeze Brahman · Xiang Ren · Yangfeng Ji · Yejin Choi · Swabha Swayamdipta -
2022 : PINTO: Faithful Language Reasoning Using Prompt-Generated Rationales »
Peifeng Wang · Aaron Chan · Filip Ilievski · Muhao Chen · Xiang Ren -
2022 : SPRINT: Scalable Semantic Policy Pre-training via Language Instruction Relabeling »
Jesse Zhang · Karl Pertsch · Jiahui Zhang · Taewook Nam · Sung Ju Hwang · Xiang Ren · Joseph Lim -
2023 Poster: SwiftSage: A Generative Agent with Fast and Slow Thinking for Complex Interactive Tasks »
Bill Yuchen Lin · Yicheng Fu · Karina Yang · Prithviraj (Raj) Ammanabrolu · Faeze Brahman · Shiyu Huang · Chandra Bhagavatula · Yejin Choi · Xiang Ren -
2023 Poster: Faith and Fate: Limits of Transformers on Compositionality »
Nouha Dziri · Ximing Lu · Melanie Sclar · Xiang (Lorraine) Li · Liwei Jiang · Bill Yuchen Lin · Sean Welleck · Peter West · Chandra Bhagavatula · Ronan Le Bras · Jena Hwang · Soumya Sanyal · Xiang Ren · Allyson Ettinger · Zaid Harchaoui · Yejin Choi -
2022 Poster: NS3: Neuro-symbolic Semantic Code Search »
Shushan Arakelyan · Anna Hakhverdyan · Miltiadis Allamanis · Luis Garcia · Christophe Hauser · Xiang Ren -
2021 Poster: SalKG: Learning From Knowledge Graph Explanations for Commonsense Reasoning »
Aaron Chan · Jiashu Xu · Boyuan Long · Soumya Sanyal · Tanishq Gupta · Xiang Ren -
2021 Poster: Gradient-based Editing of Memory Examples for Online Task-free Continual Learning »
Xisen Jin · Arka Sadhu · Junyi Du · Xiang Ren -
2021 Poster: Refining Language Models with Compositional Explanations »
Huihan Yao · Ying Chen · Qinyuan Ye · Xisen Jin · Xiang Ren -
2018 Poster: Hierarchical Graph Representation Learning with Differentiable Pooling »
Zhitao Ying · Jiaxuan You · Christopher Morris · Xiang Ren · Will Hamilton · Jure Leskovec -
2018 Spotlight: Hierarchical Graph Representation Learning with Differentiable Pooling »
Zhitao Ying · Jiaxuan You · Christopher Morris · Xiang Ren · Will Hamilton · Jure Leskovec