In this paper, we investigate the challenge of spatio-temporal video prediction, which involves generating future videos based on historical data streams. Existing approaches typically utilize external information such as semantic maps to enhance video prediction, which often neglect the inherent physical knowledge embedded within videos. Furthermore, their high computational demands could impede their applications for high-resolution videos. To address these constraints, we introduce a novel approach called Physics-assisted Spatio-temporal Network (PastNet) for generating high-quality video prediction. The core of our PastNet lies in incorporating a spectral convolution operator in the Fourier domain, which efficiently introduces inductive biases from the underlying physical laws. Additionally, we employ a memory bank with the estimated intrinsic dimensionality to discretize local features during the processing of complex spatio-temporal signals, thereby reducing computational costs and facilitating efficient high-resolution video prediction. Extensive experiments on various widely-used datasets demonstrate the effectiveness and efficiency of the proposed PastNet compared with a range of state-of-the-art methods, particularly in high-resolution scenarios. Our code is available at https://github.com/easylearningscores/PastNet.
Modeling the evolution of physical dynamics is a foundational problem in science and engineering, and it is regarded as the modeling of an operator mapping between infinite-dimensional functional spaces. Operator learning methods, learning the underlying infinite-dimensional operator in a high-dimensional latent space, have shown significant potential in modeling physical dynamics. However, there remains insufficient research on how to approximate an infinite-dimensional operator using a finite-dimensional parameter space. Inappropriate dimensionality representation of the underlying operator leads to convergence difficulties, decreasing generalization capability, and violating the physical consistency. To address the problem, we present Neural Manifold Operator (NMO) to learn the invariant subspace with the intrinsic dimension to parameterize infinite-dimensional underlying operators. NMO achieves state-of-the-art performance in statistical and physical metrics and gains 23.35% average improvement on three real-world scenarios and four equation-governed scenarios across a wide range of multi-disciplinary fields. Our paradigm has demonstrated universal effectiveness across various model structure implementations, including Multi-Layer Perceptron, Convolutional Neural Networks, and Transformers. Experimentally, we prove that the intrinsic dimension calculated by our paradigm is the optimal dimensional representation of the underlying operators. We release our code at https://github.com/AI4EarthLab/Neural-Manifold-Operators.
8 February 2024 in IEEE Transactions on Knowledge and Data Engineering
In this article, we address the issue of modeling and estimating changes in the state of the spatio-temporal dynamical systems based on a sequence of observations like video frames. Traditional numerical simulation systems depend largely on the initial settings and correctness of the constructed partial differential equations (PDEs). Despite recent efforts yielding significant success in discovering data-driven PDEs with neural networks, the limitations posed by singular scenarios and the absence of local insights prevent them from performing effectively in a broader real-world context. To this end, this article propose the universal expert module – that is, optical flow estimation component, to capture the evolution laws of general physical processes in a data-driven fashion. To enhance local insight, we painstakingly design a finer-grained physical pipeline, since local characteristics may be influenced by various internal contextual information, which may contradict the macroscopic properties of the whole system. Further, we harness currently popular neural discrete learning to unveil the underlying important features in its latent space, this process better injects interpretability, which can help us obtain a powerful prior over these discrete random variables. We conduct extensive experiments and ablations to demonstrate that the proposed framework achieves large performance margins, compared with the existing SOTA baselines.
The unusually warm sea surface temperature events known as marine heatwaves (MHWs) have a profound impact on marine ecosystems. Accurate prediction of extreme MHWs has significant scientific and financial worth. However, existing methods still have certain limitations, especially in the most extreme MHWs. In this study, to address these issues, based on the physical nature of MHWs, we created a novel deep learning neural network that is capable of accurate 10 day MHW forecasting. Our framework significantly improves the forecast ability of extreme MHWs through two specially designed modules inspired by numerical models: a coupler and a probabilistic data argumentation. The coupler simulates the driving effect of atmosphere on MHWs while the probabilistic data argumentation approaches significantly boost the forecast ability of extreme MHWs based on the idea of ensemble forecast. Compared with traditional numerical prediction, our framework has significantly higher accuracy and requires fewer computational resources. What’s more, explainable AI methods show that wind forcing is the primary driver of MHW evolution and reveal its relation with air-sea heat exchange. Overall, our model provides a framework for understanding MHWs’ driving processes and operational forecasts in the future.
The rapid development and popularity of IoT technology has reshaped the way people interact with the real world. Many researchers have attempted to build natural language interfaces for IoT platforms, but have not produced much progress in parsing natural language commands that contain multiple operations and more complex logical structures. In this paper, we propose IoT-NLI, a natural language query and control interface for popular IoT platforms, which uses hierarchical semantic parsing algorithms and directed edge-tagged graph structures to efficiently parse natural language commands input by users, enabling them to perform multiple operations contained in one complex natural language command. Experiments in three domains, agriculture, industry, and smart home, show that IoT-NLI has excellent performance and reasonable response time. Finally, a IoT-NLI application was developed on the Android platform and integrated with the AliCloud platform. It enables users to query and control devices on Android phones through chat windows similar to instant messaging software.
Human-Computer natural language interaction is helpful to reduce the operation and maintenance cost of the SCADA system, and it is necessary to solve the complex natural language interface problem that supports data query and real-time control. According to the complexity of natural language instructions, a hierarchical classification of semantic parsing algorithm is adopted. Firstly, the KWECS method is used to classify the intent of natural language instruction, then the TF-IDF keyword extraction algorithm combined with the cosine similarity is used to structure the key-value of the classified natural language instructions which was used into SCADA control intermediate language and then formally converted into actual control or query instruction. If the analysis fails, the complex control and query instruction analysis are carried out according to the classification results, structuring instruction parsing based on dependency parsing and SQL natural language parsing based on deep learning are adopted respectively to implement real-time control interface and database query interface. Our experimental results show that the proposed hierarchical classification of natural language comprehensive query and control interface can better solve the problem of human-computer natural language interaction in the SCADA system, and the accuracy of intent recognition reaches 96.5%. In more detail, the accuracy, precision, recall, and F-score of instruction parsing reach 88.47%, 90.21%, 89.48%, and 89.72% respectively. Especially, it provides more convenient interactive means for industrial and agricultural information management and control.