Over a decade ago, Microsoft introduced Parallel Data Warehouse (PDW), a massively parallel processing system to manage and query large amounts of data. Its optimizer was built by reusing SQL Server's infrastructure with minimal changes, which was an effective approach to bring cost-based query optimization quickly to PDW. Over time, learnings from production as well as architectural changes in the product (such as moving from an appliance form factor to the cloud, separation of compute and storage, and serverless components) required evolving the query optimizer in Fabric DW, the latest offering from Microsoft in the cloud data warehouse space. In this paper we describe the changes to the optimization process in Fabric DW, compare them to the earlier architecture used in PDW, and validate our new approach.
Over the past three decades, Microsoft has developed several multibillion-dollar data management products. We have now unified them into Microsoft Fabric, a comprehensive SaaS suite for data management based on a modern cloud-native design that delivers best-in-class analytics performance on CPUs. Meanwhile, Microsoft's AI investments have transformed our data centers. Hardware accelerators are now plentiful and surpass traditional CPU servers by orders of magnitude in compute, memory, and networking capabilities. Running analytics on hardware accelerators is therefore a business imperative. In this paper, we describe a multi-year effort to take Fabric to the next level by enabling analytics execution on state-of-the-art hardware accelerators. Given the rapid evolution of hardware, we have made hardware flexibility our guiding principle. We validate our architectural flexibility by demonstrating how we can run Fabric engines (e.g., Data Warehouse) on a variety of compute and network accelerators (e.g., GPUs, FPGAs, ASICs, NVLink, InfiniBand). We present our most mature implementation, a GPU-based execution engine derived from our Tensor Query Processor, and report key results from a novel data movement system leveraging NVLink and InfiniBand. This version of Fabric outperforms its CPU counterpart by over an order of magnitude across a host of production and benchmark scenarios (e.g., delivering up to 30× speedup on TPC-H 1TB). We share the lessons learned in building one of the first systems in the era of accelerated analytics.
Bitmap filtering is a technique used to accelerate queries that contain hash-based joins. Beyond simple select-project-join queries, bitmap filters can also be applied to more complex query patterns, sometimes achieving performance comparable to alternative plans that query engines may not otherwise consider. At the same time, placing bitmap filters as a post-processing step after optimization can result in suboptimal choices. In this paper, we describe how local and distributed bitmap filters are integrated in the Microsoft Fabric Data Warehouse (or Fabric DW). We highlight challenges and solutions in the spaces of optimization, scheduling, and execution. We complement our presentation with an experimental evaluation over synthetic benchmarks and real data workloads that validate our approach.