Xiaoyong Jin

Xiaoyong Jin

Auto-generated profile, based on publicly available information

Xiaoyong Jin

University of California Santa Barbara

Research Focus & Metrics

Academic Career Length

8Years (2019-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
3 January 2020 in arXiv
Verified
Preprint
Open access
PDF file
SLShiyang Li
Shiyang Li
XJXiaoyong Jin
Xiaoyong Jin
YXYao Xuan
Yao Xuan
XZXiyou Zhou
Xiyou Zhou
WCWenhu Chen
Wenhu Chen
XYXifeng Yan
Xifeng Yan
Time series forecasting is an important problem across many domains, including predictions of solar plant energy output, electricity consumption, and traffic jam situation. In this paper, we propose to tackle such forecasting problem with Transformer [1]. Although impressed by its performance in our preliminary study, we found its two major weaknesses: (1) locality-agnostics: the point-wise dot-product self-attention in canonical Transformer architecture is insensitive to local context, which can make the model prone to anomalies in time series; (2) memory bottleneck: space complexity of canonical Transformer grows quadratically with sequence length $L$, making directly modeling long time series infeasible. In order to solve these two issues, we first propose convolutional self-attention by producing queries and keys with causal convolution so that local context can be better incorporated into attention mechanism. Then, we propose LogSparse Transformer with only $O(L(\log L)^{2})$ memory cost, improving forecasting accuracy for time series with fine granularity and strong long-term dependencies under constrained memory budget. Our experiments on both synthetic data and real-world datasets show that it compares favorably to the state-of-the-art.

21 October 2019 in arXiv
Verified
Preprint
Open access
PDF file
YZYunkai Zhang
Yunkai Zhang
QJQiao Jiang
Qiao Jiang
SLShurui Li
Shurui Li
XJXiaoyong Jin
Xiaoyong Jin
XMXueying Ma
Xueying Ma
XYXifeng Yan
Xifeng Yan
Time series forecasting with limited data is a challenging yet critical task. While transformers have achieved outstanding performances in time series forecasting, they often require many training samples due to the large number of trainable parameters. In this paper, we propose a training technique for transformers that prepares the training windows through random sampling. As input time steps need not be consecutive, the number of distinct samples increases from linearly to combinatorially many. By breaking the temporal order, this technique also helps transformers to capture dependencies among time steps in finer granularity. We achieve competitive results compared to the state-of-the-art on real-world datasets.