5 June 2025 in IEEE Micro
Generative Large Language Model (LLM) applications are rapidly growing, leading to widespread deployment of expensive, power-hungry GPUs. Growing power demands of AI in the cloud industry has become a global problem [5]. Our analysis shows that LLM inference involves two distinct phases: a compute-intensive prefill phase and a memory-intensive decode phase, each with different resource needs. Running them together introduces inefficient scheduling. Furthermore, unlike prefill phase, the decode phase can run on lower-cost and lowerpower hardware. Building on these insights, we propose Splitwise, a scheduling technique that splits prefill and decode phases across different machines to achieve better throughput. Additionally, Splitwise allows phase-specific hardware optimization. By efficiently transferring request state between machines, Splitwise achieves up to 2.35× more throughput within the same power and cost budgets, or 1.4× higher throughput at 20% lower cost and same power.
22 May 2023 in IEEE Computer Architecture Letters
As modern server GPUs are increasingly power intensive, better power management mechanisms can significantly reduce the power consumption, capital costs, and carbon emissions in large cloud datacenters. This paper uses diverse datacenter workloads to study the power management capabilities of modern GPUs. We find that current GPU management mechanisms have limited compatibility and monitoring support under cloud virtualization. They have sub-optimal, imprecise, and non-intuitive implementations of Dynamic Voltage and Frequency Scaling (DVFS) and power capping. Consequently, efficient GPU power management is not widely deployed in clouds today. To address these limitations, we make actionable recommendations for GPU vendors and researchers.
1 August 2018 in Journal of Systems Architecture