25 March 2024 in IEEE Transactions on Parallel and Distributed Systems
The popularity of recommendation models and the enhanced AI processing capability of CPUs have provided massive performance opportunities to deliver satisfactory experiences to a large number of users. Unfortunately, existing recommendation model training methods fail to achieve high efficiency due to unique challenges such as dynamic shape and high parallelism. To address the above limitations, we comprehensively study the distinctive characteristics of recommendation models and discover several unexploited optimization opportunities. To exploit such opportunities, we propose AtRec , a high-performant recommendation model training engine that significantly accelerates the training process on CPUs. Specifically, AtRec presents comprehensive approach of training that employs operator-level and graph-level joint optimizations and runtime optimization. At the operator-level, AtRec identifies and optimizes the time-consuming operators, which enables further efficient graph-level optimizations. At the graph-level, AtRec conducts an in-depth analysis of the inefficiencies in several frequently used subgraphs, enables further performance improvement via eliminating redundant computations and memory accesses. In addition, to achieve better runtime performance, AtRec also identifies inefficiencies prevalent in the current scheduling and proposes runtime batching. The experiment results demonstrate that AtRec can significantly outperform state-of-the-art recommendation model training engines. We have open sourced the implementation and corresponding data of AtRec to boost research in this direction.
28 August 2024 in IEEE Transactions on Computers
Recommendation serving with deep learning models is one of the most valuable services of modern E-commerce companies. In production, to accommodate billions of recommendation queries with stringent service level agreements, high-performant recommendation serving systems play an essential role in meeting such daunting demand. Unfortunately, existing model serving frameworks fail to achieve efficient serving due to unique challenges such as 1) the input format mismatch between service needs and the model's ability and 2) heavy software contentions to concurrently execute the constrained operations. To address the above challenges, we propose RecServe , a high-performant serving system for recommendation with the optimized design of structured features and SessionGroups for recommendation serving. With structured features , RecServe packs single-user-multiple-candidates inputs by semi-automatically transforming computation graphs with annotated input tensors, which can significantly reduce redundant network transmission, data movements, and useless computations. With session group , RecServe further adopts resource isolations for multiple compute streams and cost-aware operator scheduler with critical-path-based schedule policy to enable concurrent kernel execution, further improving serving throughput. The experiment results demonstrate that RecServe can achieve maximum performance speedups of 12.3 $\boldsymbol{\times}$ and $22.0\boldsymbol{\times}$ compared to the state-of-the-art serving system on CPU and GPU platforms, respectively.