In many real-world scenarios (e.g., academic networks, social platforms), different types of entities are not only associated with texts but also connected by various relationships, which can be abstracted as Text-Attributed Heterogeneous Graphs (TAHGs). Current pretraining tasks for Language Models (LMs) primarily focus on separately learning the textual information of each entity and overlook the crucial aspect of capturing topological connections among entities in TAHGs. In this paper, we present a new pretraining framework for LMs that explicitly considers the topological and heterogeneous information in TAHGs. Firstly, we define a context graph as neighborhoods of a target node within specific orders and propose a topology-aware pretraining task to predict nodes involved in the context graph by jointly optimizing an LM and an auxiliary heterogeneous graph neural network. Secondly, based on the observation that some nodes are text-rich while others have little text, we devise a text augmentation strategy to enrich textless nodes with their neighbors' texts for handling the imbalance issue. We conduct link prediction and node classification tasks on three datasets from various domains. Experimental results demonstrate the superiority of our approach over existing methods and the rationality of each design. Our code is available at https://github.com/Hope-Rita/THLM.
Accurate traffic prediction is a crucial component of intelligent transportation systems. While deep learning models have achieved considerable success in capturing spatio-temporal dependencies, these architectures are predominantly zone-specific and struggle to generalize to data-scarce regions without extensive parameter fine-tuning. Although recent approaches leveraging Large Language Models (LLMs) offer promising zero-shot prediction capabilities, they frequently suffer from hallucination issues, which severely compromise the reliability of forecasting in real-world applications. To address these critical limitations, this paper proposes a novel Multi-modal Retrieval-Augmented Traffic Prediction (MRATP) framework. This approach captures complex spatio-temporal dynamics in a completely zone-agnostic manner and effectively mitigates LLM hallucinations by grounding predictions in robust external evidence. Specifically, we construct a scalable spatio-temporal knowledge base that structurally integrates multidimensional time information, spatial metadata, and historical traffic series. A unified multi-modal encoder is designed to project both the input queries and knowledge entries into a shared embedding space, allowing a dedicated retriever to extract the most contextually relevant historical patterns. The retrieved expert knowledge is then synthesized with the input query to form structured instructions to prompt a frozen LLM to generate highly accurate predictions. Furthermore, to optimize the retrieval process, we introduce an innovative retrieval loss that leverages the LLM's intrinsic attention mechanisms as training signals, which enables end-to-end optimization of the retriever. Extensive experiments have been conducted on two real-world datasets for traffic prediction tasks. The results demonstrate the superiority of our method in zero-shot scenarios without the need for extra parameter tuning.