Spatio-Temporal (ST) data is pervasive on the various aspects in our daily lives. By mining the ST information from the data, we are able to predict trends in numerous domains. The Transformer and one of its more recent enhancements, foundation models, have achieved a remarkable success in such ST prediction. In this article, we first survey the state of the art of Transformers-related work, then introduce the network architecture of the Transformer and summarize the improvements to adapt to the ST prediction Transformer and foundation models, including module enhancement and adjustment. Subsequently, we categorize the ST Transformer and foundation models in selected applications in some relevant domains, mainly urban transportation, climate monitoring, and motion prediction. Next, we propose an evaluation method in the ST prediction with Transformers and foundation models, list the most relevant open-source datasets, evaluation metrics and performance analysis. Finally, we discuss some future directions on the task of ST prediction with Transformer and foundation models. Relevant papers and open-source resources have been collated and are continuously updated at: https://github.com/cyhforlight/Spatio-Temporal-Prediction-Transformer-Review .

In natural language processing, event detection is a critical step in event extraction, aiming to detect the occurrences of events and categorize them. Currently, the defects of Chinese event detection based on triggers include polysemous triggers and trigger-word mismatches, which reduce the accuracy of event detection models. Therefore, event detection without triggers based on dual attention (EDWTDA), a trigger-free model that can skip the trigger identification process and determine event types directly, is proposed to fix the problems mentioned above. EDWTDA adopts a dual attention mechanism, integrating local and global attention. Local attention captures key semantic information in sentences and simulates hidden event trigger words to solve the problem of trigger-word mismatch, while global attention digs for the context of documents, fixing the problem of polysemous triggers. Besides, event detection is transformed into a binary classification task to avoid problems caused by multiple tags. Meanwhile, the sample imbalance brought about by the transformation is settled with the application of the focal loss function. The experimental results on the ACE 2005 Chinese corpus show that, compared with the best baseline model, JMCEE, the accuracy rate, recall rate, and F1-score of the proposed model increased by 3.40%, 3.90%, and 3.67%, respectively.

Compared with English named entity recognition (NER), Chinese NER faces significant challenges due to the flexible, non-standard word formation and vague word boundaries, which cause a lot of boundary ambiguity and reduce the accuracy of entity identification. To address this issue, we propose a boundary enhancement with multi-class information model (BEMCI). The model integrates multiple types of information into text embedding while enhancing the subsequent syntax-structure information. A syntactic information analysis module is designed to highlight important syntax information from three aspects, namely part-of-speech tags, syntactic constituents, and dependency relations, to analyze sentence structures. Meanwhile, an improved contextual attention mechanism, which combines contextual and syntactic information using a gate mechanism to control the weight fusion, is proposed to further enhance the model’s boundary determination. Multiple sets of experiments conducted on six general datasets show that BEMCI outperforms other baselines, achieving the best results in four of these six datasets.

The text-to-SQL task aims to convert natural language questions into corresponding SQL queries based on a given database schema. Previous models that rely on graph neural networks often struggle to accurately capture the complex grammatical relationships present in these questions, leading to poor performance when generating queries for longer requests. To address these challenges, we propose RGISQL, which integrates refined grammatical information extracted from the question and employs segmentation processing to effectively manage long queries. Additionally, RGISQL minimizes the complexity of edge embeddings by reducing the coupling within graph neural networks. By utilizing grammatical dependency trees, RGISQL is better equipped to capture the inherent structure and grammatical rules of questions. This refined grammatical information offers additional contextual and semantic cues for the model, thereby enhancing both its generalizability and interpretability. Furthermore, we dynamically assess the importance of different edges based on the graph structure, which helps reduce the coupling of edge embeddings and further improves the model’s performance. Multiple sets of experiments conducted on the Spider and Spider-Syn datasets demonstrate that RGISQL outperforms other baselines, achieving the best results in both datasets.

Code search has received significant attention in the field of computer science research. Its core objective is to retrieve the most semantically relevant code snippets by aligning the semantics of natural language queries with those of programming languages, thereby contributing to improvements in software development quality and efficiency. As the scale of public code repositories continues to expand rapidly, the ability to accurately understand and efficiently match relevant code has become a critical challenge. Furthermore, while numerous studies have demonstrated the efficacy of deep learning in code-related tasks, the mapping and semantic correlations are often inadequately addressed, leading to the disruption of structural integrity and insufficient representational capacity during semantic matching. To overcome these limitations, we propose the Functional Program Graph for Code Search (called FPGraphCS), a novel code search method that leverages the construction of functional program graphs and an early fusion strategy. By incorporating abstract syntax tree (AST), data dependency graph (DDG), and control flow graph (CFG), the method constructs a comprehensive multigraph representation, enriched with contextual information. Additionally, we propose an improved metapath aggregation graph neural network (IMAGNN) model for the extraction of code features with complex semantic correlations from heterogeneous graphs. Through the use of metapath-associated subgraphs and dynamic metapath selection via a graph attention mechanism, FPGraphCS significantly enhances its search capability. The experimental results demonstrate that FPGraphCS outperforms existing baseline methods, achieving an MRR of 0.65 and ACC@10 of 0.842, showing a significant improvement over previous approaches.