Timezone: »

 
Poster
Forecasting Future World Events With Neural Networks
Andy Zou · Tristan Xiao · Ryan Jia · Joe Kwon · Mantas Mazeika · Richard Li · Dawn Song · Jacob Steinhardt · Owain Evans · Dan Hendrycks

Wed Nov 30 09:00 AM -- 11:00 AM (PST) @ Hall J #1022
in Poster Session 3 »

Forecasting future world events is a challenging but valuable task. Forecasts of climate, geopolitical conflict, pandemics and economic indicators help shape policy and decision making. In these domains, the judgment of expert humans contributes to the best forecasts. Given advances in language modeling, can these forecasts be automated? To this end, we introduce Autocast, a dataset containing thousands of forecasting questions and an accompanying news corpus. Questions are taken from forecasting tournaments, ensuring high quality, real-world importance, and diversity. The news corpus is organized by date, allowing us to precisely simulate the conditions under which humans made past forecasts (avoiding leakage from the future). Motivated by the difficulty of forecasting numbers across orders of magnitude (e.g. global cases of COVID-19 in 2022), we also curate IntervalQA, a dataset of numerical questions and metrics for calibration. We test language models on our forecasting task and find that performance is far below a human expert baseline. However, performance improves with increased model size and incorporation of relevant information from the news corpus. In sum, Autocast poses a novel challenge for large language models and improved performance could bring large practical benefits.

Keywords: question answering · retrieval · Forecasting · Epistemics
[ Paper]  [ Poster]  [ OpenReview] 

Author Information

Andy Zou (CMU, Carnegie Mellon University)
Tristan Xiao (University of California, Berkeley)
Ryan Jia (Apple (current) / UC Berkeley (until May 2022))
Joe Kwon (Massachusetts Institute of Technology)
Mantas Mazeika (University of Illinois Urbana-Champaign)
Richard Li (University of California, Berkeley)
Dawn Song (UC Berkeley)
Jacob Steinhardt (UC Berkeley)
Owain Evans (University of Oxford)
Dan Hendrycks (Center for AI Safety)

More from the Same Authors