We present a bandwidth- and energy-efficient, hybrid ray tracing and rasterizer architecture for tile-based mobile GPU. In order to successfully commercialize mobile System on Chip (SoC), including ray tracing hardware solution, effective integration with rasterizer based on OpenGL---ES is indispensable for the performance and compatibility reason. Thus, recently, the traditional rasterizer-ray tracing hybrid approach is revisited to achieve this goal. The key factor for hybrid rendering is to reflect the fundamental principle of tile-based rendering on integrating the ray tracing hardware and mobile GPUs. Consequently, we propose a new architecture for hybrid rendering by combining new three features such as extended tile binning unit, tile prefetch, and per-tile power control. Simulation results show that our architecture is a potentially versatile solution for future mobile GPUs in low-energy devices because it provides as much as 31.7% better G-buffer bandwidth utilization and is up to 2.18 times better performance per unit energy compared to the ray tracing hardware-only solution.
Generative artificial intelligence is widely used for various tasks such as language translation and art creation. Generative inference employs the key and value tensors to encode relational information among the tokens in the input and output sequences. Most of the existing generative inference frameworks use KV caching (KVC), which caches the key and value tensors (i.e., the KV cache) to avoid recomputing the key and value tensors for each of the processed tokens. Despite the widespread use of KVC, in-depth characterization of KVC with tensor offloading, which enables generative inference with a single GPU, remains yet to be explored. To bridge this gap, this work presents an in-depth characterization study of KVC, which demonstrates that KVC makes a sub-optimal trade-off between computations and communications in the context of generative inference with tensor offloading. Guided by the characterization results, we propose a novel caching technique called activation sequence caching (ASC) for high-throughput and memory-efficient generative inference with a single GPU. ASC is designed to significantly reduce the memory usage and communication overheads of generative inference at the cost of the increased computational complexity. We provide an analytical analysis of the three caching techniques (i.e., zero caching, KVC, and ASC) for generative inference in terms of the memory usage, communication overheads, and computational complexity. Our quantitative evaluation demonstrates the effectiveness of ASC in that ASC significantly (e.g., 78.3% higher throughput) outperforms a state-of-the-art generative inference framework (i.e., FlexGen) that implements KVC across various model sizes, input and output sequence lengths, and GPUs.
With the rapid growth in memory demands, the slowdown of DRAM scaling, and the DRAM price fluctuations, DRAM has become one of the critical resources in cloud computing systems and datacenters. The compressed memory swap (CMS) is a promising technique that improves the effective memory capacity of the underlying computer system by compressing and storing a subset of pages in memory instead of the disk swap. While prior works have extensively investigated resource management techniques for workload consolidation, they lack the capability of dynamically allocating cores, memory, andCMSto the consolidated applications in a controlled and efficient manner. To bridge this gap, this work presents the in-depth characterization of the impact of cores, memory, and CMS on the QoS and throughput of the consolidated latency-critical (LC) and batch applications. Guided by the characterization results, we propose COSMOS, a software-based runtime system for coordinated management of cores, memory, and CMS for QoS-aware and efficient workload consolidation for memory-intensive applications. COSMOS dynamically collects the runtime data from the consolidated applications and the underlying system and allocates the resources to the consolidated applications in a way that achieves high throughput with strong QoS guarantees. Our quantitative evaluation based on a real system and widely-used memory-intensive benchmarks demonstrates the effectiveness of COSMOS in that it robustly satisfies the QoS and achieves high throughput across all the evaluatedworkload mixes and scenarios and significantly reduces the number of explored system states.