NVMe over Fabrics (NVMeoF) is a next-generation storage disaggregation protocol tailored for high-performance data centers utilizing NVMe SSDs and interconnected networks. Its performance advantages stem from leveraging multiple deep queue pairs to enable parallel PDUs transfers. However, as NVMe SSDs and network infrastructure continue to advance, achieving near-ideal performance in NVMeoF systems becomes increasingly challenging. This paper investigates key performance limitations in the native NVMeoF queue design, particularly focusing on issues related to the unified read-write communication pipeline, as well as intra- and inter-queue I/O performance interference. To address these challenges, we propose AEQ, a co-designed NVMeoF queue pair architecture aimed at enhancing the efficiency of data and command transmission for remote storage access. Specifically, we introduce AQ-RI, a switchable queue design that optimizes differentiated read-write request processing; EQ-RI, an end-to-end packet processing system with dedicated I/O pipelines for distinct application types; and AQ-PI, a PDU isolation mechanism that ensures intra-queue I/O isolation. Experimental results show that AEQ significantly enhances high-concurrency performance in NVMeoF storage networks, while also improving both latency and throughput quality of service.

NVMe-over-Fabrics (NVMeoF) is expected to have high-performance and be highly scalable for disaggregating NVMe SSDs to High-Speed Network (HSN)-attached storage servers, thus the aggregated NVMe SSDs in storage servers can be elastically allocated to remote host servers for better utilization. However, due to the well-known connection scalability issue of RDMA NICs (RNICs), RDMA-enabled HSN can only provide a limited scale of performant Queue Pairs (QPs) for NVMeoF I/O queues to transfer capsule and data between the storage server and remote host servers. However, in current NVMeoF implementations, multiplexing multiple NVMeoF I/O queues onto a single RNIC QP is not supported yet. In this paper, we investigate how NVMeoF capsule and data transfers are performed efficiently over HSN with a limited number of RNIC QPs, and propose SPANoF, a Scalable and Performant Architecture for NVMe-over-Fabrics. SPANoF dissolves the intrinsic one-to-one mapping relationship between NVMeoF I/O queues and RNIC QPs, allocates a dedicated send-list for each NVMeoF I/O queue rather than for each RNIC QP, transfers NVMeoF capsules and data in send-lists with a QP-centric manner to remove lock-contention overhead, and polls for transfer completion notifications to remove interrupt-caused context switch overhead. We implemented SPANoF in the Linux kernel and evaluated it by the FIO benchmarks. Our experimental results demonstrate that SPANoF can avoid the performance collapses for commercial RNICs with a limited number of performant QPs and avoid the system crash for domain-specific RNICs with only limited-scale available QPs. Compared with the native NVMeoF implementation in Linux kernel, SPANoF can saturate an RNIC of the storage server with only three RNIC QPs of the remote host server. Compared with lock-based QP-sharing mechanisms, SPANoF improves bandwidth by up to 1.55× under 64 KB sequential write requests, improves throughput by up to 4.18× and reduces the average latency by 28.31% under 4 KB random read requests.