Alexander Evgenyevich Arkhipov

Prof. Alexander Evgenyevich Arkhipov

avatar
Co-authors
avatar
Following
avatar
Followers

Research Focus & Metrics

Explore full metrics on Scilit

Academic Career Length

2Years (2025-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
25 July 2025 in Applied Sciences
Verified
Journal Article
Open access
PDF file
Tianfa LiTianfa Li
Tianfa Li
JPJingshan Pan
Jingshan Pan
SMSiwei Ma
Siwei Ma
Aleksandr RaikovAleksandr Raikov
Aleksandr Raikov
AAAlexander Evgenyevich Arkhipov
Alexander Evgenyevich Arkhipov
LLMs are trained using many thousands of GPUs in well-known conventional models. It is necessary to address numerous issues in the training process, such as manual data collection organization, data parallel, model parallel, evaluation, testing, deployment, transferring large data streams, detecting errors, ongoing maintenance, and project management. A team of dozens of engineers is required to handle system problems in the training process. Therefore, it is time-consuming and expensive to build an efficient and fault-tolerant system based on Kubernetes. This paper develops SimpleScale for building LLMs based on FSDP and Slurm, which is a simple and efficient training system that includes the training agent, the efficient parallel strategy, the optimal step of checkpoint, and so on. Using the proposed system enables us to significantly accelerate the process of building the LLM without incurring substantial time spent on various manual issues. The proposed 1024-GPU cluster was tested on TinyLlama, which has 1.1 billion parameters and 300 billion tokens. For example, utilizing a 16-H100 GPU cluster accelerated the traditional TinyLlama training time costs from 89.05 days to 39.05 days. In the future, the project team plans to integrate Flash-Attention3, aiming for an MFU of more than 60% while maintaining the acceleration achieved in the present work.