12 December 2025 in ACM Transactions on Architecture and Code Optimization
Graph Neural Networks (GNNs) have become essential for graph-based applications. Existing GNN training frameworks primarily adopt sampling-based mini-batch approach, which samples vertex neighborhoods to construct subgraphs for training iteratively, but faces a memory-accuracy tradeoff. To address this, naive micro-batch training with gradients accumulation can be applied, yet introduces severe redundancy from repeated neighbor sampling. Recent micro-batch training methods overcome this by adopting batch-level graph partitioning strategies that partition full-batch subgraph sampled from all target vertices to generate micro-batches. Nevertheless, these approaches struggle to balance partitioning overhead with redundancy. This article proposes PDGNN, a system designed for efficient micro-batch GNN training on single machine, reducing both training time and memory consumption. First, we propose a degree-pruning-based graph partitioning method that efficiently processes on simplified and sparsified multi-level bipartite graphs by removing high-degree vertices, significantly reducing partitioning overhead. Second, we propose a dependency-aware redundancy matching and elimination technique that maximizes inter-micro-batch redundancy matching, and eliminates redundant computation by embedding caching and reuse while maintaining model accuracy. Experiments on real-world datasets demonstrate that PDGNN substantially improves the efficiency of large-scale GNN training, achieving up to 3.5× speedup in epoch time, 46% lower peak GPU memory consumption, and 64% reduced peak CPU memory consumption compared to state-of-the-art batch-level training methods.
24 November 2025 in IEEE Transactions on Parallel and Distributed Systems
Graph Neural Networks (GNNs) with learnable vertex embeddings enable models to infer rich, task-specific representations even when vertex features are sparse, noisy, or missing. In large-scale multi-GPU training, dynamically updated embeddings, often orders of magnitude larger than model parameters, severely degrade training efficiency. Specifically, loading remote embeddings and synchronizing their gradients collectively account for over 90% of per-iteration time. Traditional caching and parallelism approaches, designed for static embeddings or model parameters alone, are ineffective at mitigating this “data wall” of embedding-related transfers. To address this, we begin with a detailed analysis of vertex access patterns over training iterations and find that infrequently sampled vertices, despite incurring the majority of embedding-loading latency, undergo very few updates, making their embeddings ideal candidates for staleness reuse. Driven by this, we propose GLPilot, a novel system that mitigates embedding-related bottlenecks. GLPilot introduces a staleness-bounded embedding buffering mechanism to reduce remote fetches and a local gradient aggregation technique to minimize redundant communications during synchronization. Additionally, GLPilot utilizes an on-GPU cache for keeping mostly updated embeddings to alleviate CPU-GPU data transfer bottlenecks. Our evaluations on a 32-GPU cluster using two popular GNN models, three datasets and two optimizers demonstrate that GLPilot consistently achieves 1.28–1.93× per-epoch training speedups, in comparison with two strong baselines such as DGL and P3, while maintaining comparable model accuracy.