With the continued digitization of social and industrial processes, diverse and increasingly massive volumes of spatial-temporal data are being accumulated， which enable the use of data mining to fuel important applications. Thus, neural network-based techniques have been developed to capture spatial and temporal dependencies in spatial-temporal data. Most recently, the advances in generative techniques, including large language models, masked autoencoders, sequence-to-sequence models, diffusion models, and others, have led to their increased use in spatial-temporal data mining, thereby driving new advances in spatial-temporal data mining. This paper describes the general types of spatial-temporal data, all kinds of spatial-temporal data instances, and generative techniques, and it proposes a spatial-temporal data mining pipeline. Further, it delivers a structured overview of generative techniques for spatial-temporal data mining, grounded in a novel taxonomy. Moreover, by outlining promising research avenues enabled by generative techniques, the paper seeks to accelerate advances in spatial-temporal data mining.

Physics-informed machine learning has emerged as a promising approach for modeling physical systems. However, real-world applications often face significant challenges due to the limitations of partial observations and inaccuracies in governing partial differential equations (PDEs). In this work, we propose a novel physics-informed machine learning method named Physics-Informed method based on Inaccurate PDEs and Partial Observation (PIPO), to overcome the above two challenges under the real-world scenario, which aims to address the problems of super-resolution and forecasting simultaneously in physical systems characterized by partial observations and inaccurate PDEs. The proposed method is motivated by two key considerations. First, despite the inherent inaccuracy of PDEs, the differential terms (such as diffusion and advection terms) contain valuable information that can effectively reduce the hypothesis space, thereby enhancing the model's generalization capability. Second, while the data only provides partial observations, it offers crucial supervised constraints at the observed points. These constraints not only facilitate model optimization but also help prevent the degeneracy of PDE loss, where the PDE loss could yield multiple solutions. PIPO integrates an interpolator, encoder, forecaster, decoder, and parameters learner, which are jointly optimized using data loss and PDE losses to reconstruct high-resolution states and forecast future states using only partial observation data and inaccurate PDEs. We leverage the proposed PIPO method to address a real-world problem in air pollutant concentration fields and wind fields, specifically PM2.5 transport dynamics, which are governed by the advection-diffusion equation with unknown diffusion coefficients and the source term. The results in super-resolution reconstruction, forecasting, and multi-hour forecasting highlight the effectiveness of PIPO in capturing complex spatial and temporal dynamics despite the limitations of partial observations and inaccurate PDEs.

Deep learning-based surrogate models have demonstrated remarkable advantages over classical solvers in terms of speed, often achieving speedups of 10 to 1000 times over traditional partial differential equation (PDE) solvers. However, a significant challenge hindering their widespread adoption in both scientific and industrial domains is the lack of understanding about their prediction uncertainties, particularly in scenarios that involve critical decision making. To address this limitation, we propose a method that integrates efficient and precise uncertainty quantification into a deep learning-based surrogate model. Our method, termed Latent Evolution of PDEs with Uncertainty Quantification (LE-PDE-UQ), endows deep learning-based surrogate models with robust and efficient uncertainty quantification capabilities for both forward and inverse problems. LE-PDE-UQ leverages latent vectors within a latent space to evolve both the system's state and its corresponding uncertainty estimation. The latent vectors are decoded to provide predictions for the system's state as well as estimates of its uncertainty. In extensive experiments, we demonstrate the accurate uncertainty quantification performance of our approach, surpassing that of strong baselines including deep ensembles, Bayesian neural network layers, and dropout. Our method excels at propagating uncertainty over extended auto-regressive rollouts, making it suitable for scenarios involving long-term predictions. Our code is available at: https://github.com/AI4Science-WestlakeU/le-pde-uq.

Antibodies are Y-shaped proteins that protect the host by binding to specific antigens, and their binding is mainly determined by the Complementary Determining Regions (CDRs) in the antibody. Despite the great progress made in CDR design, existing computational methods still encounter several challenges: 1) poor capability of modeling complex CDRs with long sequences due to insufficient contextual information; 2) conditioned on pre-given antigenic epitopes and their static interaction with the target antibody; 3) neglect of specificity during antibody optimization leads to non-specific antibodies. In this paper, we take into account a variety of node features, edge features, and edge relations to include more contextual and geometric information. We propose a novel Relation-Aware Antibody Design (RAAD) framework, which dynamically models antigen-antibody interactions for co-designing the sequences and structures of antigen-specific CDRs. Furthermore, we propose a new evaluation metric to better measure antibody specificity and develop a contrasting specificity-enhancing constraint to optimize the specificity of antibodies. Extensive experiments have demonstrated the superior capability of RAAD in terms of antibody modeling, generation, and optimization across different CDR types, sequence lengths, pre-training strategies, and input contexts.

This paper focuses on the integration of generative techniques into spatial-temporal data mining, considering the significant growth and diverse nature of spatial-temporal data. With the advancements in RNNs, CNNs, and other non-generative techniques, researchers have explored their application in capturing temporal and spatial dependencies within spatial-temporal data. However, the emergence of generative techniques such as LLMs, SSL, Seq2Seq and diffusion models has opened up new possibilities for enhancing spatial-temporal data mining further. The paper provides a comprehensive analysis of generative technique-based spatial-temporal methods and introduces a standardized framework specifically designed for the spatial-temporal data mining pipeline. By offering a detailed review and a novel taxonomy of spatial-temporal methodology utilizing generative techniques, the paper enables a deeper understanding of the various techniques employed in this field. Furthermore, the paper highlights promising future research directions, urging researchers to delve deeper into spatial-temporal data mining. It emphasizes the need to explore untapped opportunities and push the boundaries of knowledge to unlock new insights and improve the effectiveness and efficiency of spatial-temporal data mining. By integrating generative techniques and providing a standardized framework, the paper contributes to advancing the field and encourages researchers to explore the vast potential of generative techniques in spatial-temporal data mining.

Inverse design, where we seek to design input variables in order to optimize an underlying objective function, is an important problem that arises across fields such as mechanical engineering to aerospace engineering. Inverse design is typically formulated as an optimization problem, with recent works leveraging optimization across learned dynamics models. However, as models are optimized they tend to fall into adversarial modes, preventing effective sampling. We illustrate that by instead optimizing over the learned energy function captured by the diffusion model, we can avoid such adversarial examples and significantly improve design performance. We further illustrate how such a design system is compositional, enabling us to combine multiple different diffusion models representing subcomponents of our desired system to design systems with every specified component. In an N-body interaction task and a challenging 2D multi-airfoil design task, we demonstrate that by composing the learned diffusion model at test time, our method allows us to design initial states and boundary shapes that are more complex than those in the training data. Our method generalizes to more objects for N-body dataset and discovers formation flying to minimize drag in the multi-airfoil design task. Project website and code can be found at https://github.com/AI4Science-WestlakeU/cindm.