30 June 2025 in ACM Transactions on Architecture and Code Optimization
MPI tracing tools is essential to collect the communication events and performance metrics of large-scale programs for further performance analysis and optimization. However, toward the exascale era, the performance and storage overhead for tracing becomes extremely prohibitive that significantly disturbs the original execution of MPI programs, leading to distorted tracing data and thus mislead analysis results. Although process sampling can effectively reduce the tracing overhead, it can easily miss important execution information that is necessary for subsequent performance analysis. In this article, we propose SimTrace , a scalable MPI tracing tool with novel spatial and temporal sampling strategies that exploits the similarity among MPI processes to achieve both low tracing overhead as well as obtain sufficient tracing information. The experimental results demonstrate that SimTrace can significantly reduce the MPI tracing overhead compared to the state-of-the-art tracing tools, meanwhile enabling effective analysis to guide performance optimization of large-scale programs.
25 March 2024 in IEEE Transactions on Parallel and Distributed Systems
The popularity of recommendation models and the enhanced AI processing capability of CPUs have provided massive performance opportunities to deliver satisfactory experiences to a large number of users. Unfortunately, existing recommendation model training methods fail to achieve high efficiency due to unique challenges such as dynamic shape and high parallelism. To address the above limitations, we comprehensively study the distinctive characteristics of recommendation models and discover several unexploited optimization opportunities. To exploit such opportunities, we propose AtRec , a high-performant recommendation model training engine that significantly accelerates the training process on CPUs. Specifically, AtRec presents comprehensive approach of training that employs operator-level and graph-level joint optimizations and runtime optimization. At the operator-level, AtRec identifies and optimizes the time-consuming operators, which enables further efficient graph-level optimizations. At the graph-level, AtRec conducts an in-depth analysis of the inefficiencies in several frequently used subgraphs, enables further performance improvement via eliminating redundant computations and memory accesses. In addition, to achieve better runtime performance, AtRec also identifies inefficiencies prevalent in the current scheduling and proposes runtime batching. The experiment results demonstrate that AtRec can significantly outperform state-of-the-art recommendation model training engines. We have open sourced the implementation and corresponding data of AtRec to boost research in this direction.