The prediction of urban crowds is crucial not only to traffic management but also to studies on the city-level social phenomena, such as energy consumption, urban growth, city planning, and epidemic prevention. The challenges of accurately predicting crowd flow come from the non-linear spatial-temporal dependence of crowd flow data, periodic laws, such as daily and weekly periodicity, and external factors, such as weather and holidays. It is even more challenging for most existing short-term prediction models to make an accurate long-term prediction. In this paper, we propose a novel patched Transformer-based sequence-to-sequence model, called MultiSize Patched Spatial-Temporal Transformer Network (MSP-STTN), to incorporate rich and unified context modeling via a self-attention mechanism and global memory learning via a cross-attention mechanism for short-and long-term grid-based crowd flow prediction. In particular, a multisize patched spatial-temporal self-attention Transformer is designed to capture cross-space-time and cross-size contextual dependence of crowd data. The same structured cross-attention Transformer is developed to adaptively learn a global memory for long-term prediction in a responding-to-a-query style without error accumulation. In addition, a categorized space-time expectation is proposed as a unified regional encoding with temporal and external factors and is used as a base prediction for stable training. Furthermore, auxiliary tasks are introduced for promoting feature encoding and leveraging feature consistency to assist in the main prediction task. The experimental results reveal that MSP-STTN is competitive with the state of the art for one-step and multi-step short-term prediction within several hours and achieves practical long-term crowd flow prediction within one day on real-world grid-based crowd data sets TaxiBJ, BikeNYC, and CrowdDensityBJ. Our code and data are available at https://github.com/xieyulai/ MSP-STTN.

A novel technique is proposed for joint multi-impairment optical performance monitoring (OPM) with bit-rate and modulation format identification (BR-MFI) in next-generation heterogeneous optic communication networks by convolution neural network (CNN)-based deep multi-task learning (MTL) on asynchronous delay-tap sampling (ADTS) phase portraits. Instead of treating the monitoring and identification tasks as separate problems, a novel MTL technique is used to joint optimization of them utilizing the ability of feature extraction and feature sharing. Compared with principal component analysis (PCA)-based pattern recognition algorithm, CNN-based MTL achieves the better accuracies and has a shorter processing time (~56ms). The combination signals of three modulation formats and two bit rates under various impairments are used in numerical simulation. For OPM, the results show monitoring of optical signal-to-noise ratio (OSNR), chromatic dispersion (CD), and differential group delay (DGD) with root-mean-square (RMS) error of 0.73 dB, 1.34 ps/nm, and 0.47 ps, respectively. Similarly, for BR-MFI, even in the case of limited training data, 100% accuracies can be achieved. Additionally, the effects of training data size, task weights, and model structure on CNN-based MTL performance are comprehensively studied. The proposed technique can intelligently analyze the signals of future heterogeneous optic communication networks, and the analysis results are helpful for better management of optical networks.

Graph convolution networks (GCNs) have drawn attention for skeleton‐based action recognition. They have achieved remarkable performance by adaptively learning spatial features of human action dynamics. However, the existing methods are limited in temporal sequence modelling of human actions. To give adequate consideration to temporal factors in action modelling, a novel temporal‐enhanced graph convolution network is presented. First, a Causal Convolution layer is introduced to ensure no future information leakage at each time step for keeping ordering information of inputs. Second, a novel cross‐spatial‐temporal graph convolution layer that extends an adaptive graph from the spatial to the temporal domain to capture local cross‐spatial‐temporal dependencies among joints is presented. Third, a temporal attention layer is designed to enhance the modelling capability of long‐range temporal dependencies, helping the network to directly focus on important time steps. Experimental results on three large‐scale datasets, NTU‐RGB + D, Kinetics‐Skeleton, and UAV‐Human, indicate that the authors’ network achieves accuracy improvement with better generalisation capability over previous methods. The authors’ code and data are available at https://github.com/xieyulai/TE‐GCN .

We propose a novel feature fusion based multi-task convolutional neural network (ConvNet) for simultaneous bit-rate and modulation format identification (BR-MFI) and optical performance monitoring (OPM) in heterogeneous fiber-optic networks. The proposed multi-task ConvNet fuses the intermediate layers through the convolutional operation and then trains multi-task losses on the fused feature. In addition to traditional multi-task ConvNet’s ability of the feature extraction and sharing, our multi-task ConvNet is able to capture both global and local information of phase portraits and has good performance on OPM and BR-MFI tasks in a short processing time ( 51 ms). The simulation results of six signals (consisted of two bit-rates and three modulation formats) demonstrate the root-mean-square (RMS) errors of the optical signal-to-noise ratio (OSNR), chromatic dispersion (CD) and differential group delay (DGD) are 0.81 dB, 1.52 ps/nm and 0.32 ps, respectively. Meanwhile, the 100% classification accuracy can be obtained for BR-MFI. Besides, the effects of the fused feature shape, the location of feature extracted for fusion, the transmitter variations and fiber nonlinearity on the performance of the proposed technique are thoroughly investigated.

Dense video captioning aims to detect all events of an uncropped video and generate corresponding textual captions for each event. Multi-modal information is essential to improve the performance of this task, but the existing methods mainly rely on the single visual or dual audio-visual modal input, while completely ignoring the text modal input (subtitle). Since the text data has a similar data representation as video caption words, it is conducive to the performance improvement of video captioning. In this paper, we propose a novel framework, called the multi-stage fusion transformer network (MS-FTN), to realize multi-modal dense video captioning by fusing the text, the audio, and the visual features in stages. We present a multi-stage feature fusion encoder that first fuses audio and visual modalities at a lower level and then fuses them with a global-shared text representation at a higher level to generate a set of multi-modal complementary context features. In addition, an anchor-free event proposal module is proposed to efficiently generate a set of event proposals without the complex anchor calculation. Extensive experiments on the subsets of the ActivityNet Captions dataset show that our proposed MS-FTN achieves superior performance and efficient computation. Moreover, the ablation studies demonstrate that the global-shared text representation is more suitable for multi-modal dense video captioning. Our code and data are available at https://github.com/xieyulai/GS-MS-FTN .