9 July 2024 in IEEE Micro
The explosive arrival of OpenAI’s ChatGPT has fueled the globalization of large language model (LLM), which consists of billions of pretrained parameters that embodies the aspects of syntax and semantics. HyperAccel introduces latency processing unit (LPU), a latency-optimized and highly scalable processor architecture for the acceleration of LLM inference. LPU perfectly balances the memory bandwidth and compute logic with streamlined dataflow to maximize performance and efficiency. LPU is equipped with expandable synchronization link (ESL) that hides data synchronization latency between multiple LPUs. HyperDex complements LPU as an intuitive software framework to run LLM applications. LPU achieves 1.25 ms/token and 20.9 ms/token for 1.3B and 66B model, respectively, which is 2.09× and 1.37× faster than the GPU. LPU, synthesized using Samsung 4nm process, has total area of 0.824 mm 2 and power consumption of 284.31 mW. LPU-based servers achieve 1.33× and 1.32× energy efficiency over NVIDIA H100 and L4 servers, respectively.
6 March 2026 in IEEE Journal of Solid-State Circuits
The proliferation of large language models (LLMs) as cross-domain foundation models is fueled by aggressive scaling in both parameter counts and inference-time computation. The emergence of sophisticated reasoning models further accelerates this trend, demanding longer context windows and escalating the computational and memory burdens of inference. A fundamental challenge arises from the bimodal nature of LLM inference, which consists of a compute-bound prefill phase and a memory-bound decode phase. This duality creates a significant performance bottleneck for conventional architectures such as GPUs and neural processing units (NPUs), which are typically optimized for only one phase, leading to severe underutilization of resources in the other. This article introduces Adelia, a novel LLM inference accelerator designed to resolve this challenge by co-optimizing for both memory and compute efficiency. Adelia features a streamlined dataflow that aligns sustained compute throughput with available external memory bandwidth, maximizing utilization across both inference phases. Its architecture features powerful matrix and vector execution engines (VXEs) fed by a systolic parameter path (SPP) interconnect, which efficiently multicasts model parameters and key–value (KV) caches for spatial reuse, all orchestrated by an RISC control processor (RCP) that provides dynamic instruction dispatch and enables dual-mode parallelism. By holistically addressing the bimodal bottleneck, Adelia demonstrates high performance and scalable efficiency during inference. Fabricated in Samsung’s 4-nm process, Adelia achieves a high throughput of 512.6 Tokens/s on Phi-3-mini with an end-to-end runtime of 5.993 s and 65.72 Tokens/s on LLaMA-30B with a runtime of 46.741 s.