Timezone: »

 
Poster
Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting
Shiyang Li · Xiaoyong Jin · Yao Xuan · Xiyou Zhou · Wenhu Chen · Yu-Xiang Wang · Xifeng Yan

Thu Dec 12 10:45 AM -- 12:45 PM (PST) @ East Exhibition Hall B + C #104
in Applications -- Time Series Analysis »

Time series forecasting is an important problem across many domains, including predictions of solar plant energy output, electricity consumption, and traffic jam situation. In this paper, we propose to tackle such forecasting problem with Transformer. Although impressed by its performance in our preliminary study, we found its two major weaknesses: (1) locality-agnostics: the point-wise dot- product self-attention in canonical Transformer architecture is insensitive to local context, which can make the model prone to anomalies in time series; (2) memory bottleneck: space complexity of canonical Transformer grows quadratically with sequence length L, making directly modeling long time series infeasible. In order to solve these two issues, we first propose convolutional self-attention by producing queries and keys with causal convolution so that local context can be better incorporated into attention mechanism. Then, we propose LogSparse Transformer with only O(L(log L)^2) memory cost, improving forecasting accuracy for time series with fine granularity and strong long-term dependencies under constrained memory budget. Our experiments on both synthetic data and real- world datasets show that it compares favorably to the state-of-the-art.

Keywords: Predictive Models · Attention Models · Deep Learning · Applications -> Time Series Analysis; Deep Learning
[ Paper]  [ Poster] 

Author Information

Shiyang Li (UCSB)
Xiaoyong Jin (UCSB)
Yao Xuan (University of California, Santa Barbara)
Xiyou Zhou (UC Santa Barbara)
Wenhu Chen (University of California, Santa Barbara)
Yu-Xiang Wang (UC Santa Barbara)
Xifeng Yan (UCSB)

More from the Same Authors