Dong Huang

Mr. Dong Huang


Research Focus & Metrics

Academic Career Length

1Year (2026-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
13 October 2026 in Applied Sciences
Verified
Journal Article
Open access
PDF file
LZLanxin Zhao
Lanxin Zhao
DHDong Huang
Dong Huang
WGWanrong Gao
Wanrong Gao
Jianbin FangJianbin Fang
Jianbin Fang
LLM-based language services (e.g., summarization, translation, and question answering) require low-latency and cost-effective inference, yet GPU resources may be costly, unavailable, or difficult to integrate in many deployments. Many-core CPUs provide a flexible alternative, but achieving efficient inference requires optimizations that match the distinct computational characteristics of the Prefill and Decode stages. This paper presents a stage-aware optimization approach for Llama 3 inference on two Phytium many-core processors (FT-2000+/64 and FT-1500A). Fine-grained profiling shows that matrix-vector multiplication (MVM) in the Decode stage accounts for 99.4% of Decode time, whereas Prefill exposes additional token-level parallelism. Accordingly, in Prefill we reorganize multiple token-wise MVM operations into matrix–matrix multiplication and select the GEMM implementation by micro-batch granularity, including a blocked C GEMM (up to 7.8× over token-wise MVM) and a LibShalom-based tile path. In Decode, we apply OpenMP-based intra-kernel parallelism to MVM, RMSNorm, Softmax, and attention-head computation. Experiments on representative GLUE workloads show that the self-kernel implementation improves time per output token (TPOT) by 1.67× at two threads (2.80× at eight). A complete baseline study then shows that a tuned BLAS GEMV accelerates the dominant Decode MVM by a further 3.1×; following the stage-aware selection principle, the final configuration reaches a TPOT of 141.9 ms—a 17.2× end-to-end speedup (7.05 tokens/s per 16-core node)—with scaling that continues to all cores. A kernel-level scalability study on Phytium FT-2000+/64 (Decode MVM up to 8.4× at eight threads) further shows that saturation beyond eight threads is a property of the kernel implementation rather than of the platform. These results demonstrate that stage-aware characterization and per-stage implementation selection can substantially improve single-request Llama 3 inference efficiency on many-core CPUs, and provide guidance for CPU-side optimization in language-service deployments.