2 January 2023 in IEEE/ACM Transactions on Networking
Frequency estimation and top-k flows identification are fundamental problems in network traffic measurement. Sketch, as a basic probabilistic data structure, has been extensively investigated and used in different management applications. However, few of them is suitable for both estimating frequency and finding top-k flows due to the unbalanced distribution of real-world network streams. By introducing a pre-filtering stage to isolate elephant and mice flows, the recently proposed Augmented Sketch (ASketch) significantly improves accuracy for both tasks. However, it suffers from serious performance degradation because of frequent flow exchanges. In this paper, we propose Cuckoo Counter (CC), an adaptive structure that consists of several buckets organized in a specific way. The size of the entry in each bucket is carefully designed to match the actual distribution of streams. During processing, CC hashes a flow to buckets and uses the idea of cuckoo hashing to relocate the flow if an overflow or collision happens, which contributes to fully utilizing memory. Therefore, the replacement strategy helps CC precisely record elephant flows and cover more mice flows, and also guarantees the throughput. Extensive experimental results show that CC has the highest (Freq.) accuracy, excellent (Heavy hitter / change) accuracy, highest (Top-k) precision, and competitive throughput compared to the state-of-the-art. Specifically, CC improves the throughput and accuracy by around 1 and 2 orders of magnitude respectively compared to the well-known ASketch.
27 February 2026 in IEEE Transactions on Networking
In-network aggregation (INA) has emerged as a key technology to alleviate communication bottlenecks in large-scale distributed training, but its performance is often hindered by suboptimal routing. Existing INA-aware routing algorithms suffer from certain limitations: they either lack a global, multi-job coordination mechanism, or operate on incomplete network models that ignore key hardware constraints such as switch processing capacity. These deficiencies lead to network congestion and inefficient resource utilization, ultimately undermining the full potential of INA. To address these challenges, we present INARouting, a novel framework that holistically solves the multi-job hierarchical aggregation routing problem. We propose TINA, a hierarchical aggregation protocol that supports multi-job in-network aggregation. To address different deployment scenarios, we develop two variants: INARouting-Opt that provides optimal solutions for moderate-scale networks, and INARouting-Relax, a fast and effective heuristic using LP-relaxation and a greedy score-based rounding algorithm for large-scale deployments. Through extensive experiments on various scales of Fat-Tree and Spine-Leaf topologies, we demonstrate that INARouting significantly outperforms state-of-the-art methods. INARouting- Opt achieves provably optimal solutions, reducing average job completion time by up to 56% compared to existing methods. Meanwhile, INARouting-Relax outperforms existing algorithms while being 5× faster in solving time, enabling efficient routing in large-scale, dynamic environments.
26 December 2024 in IEEE Transactions on Knowledge and Data Engineering
In distributed data stream mining, we abstract a MIMO scenario where a stream of m ultiple i tems is mined by m ultiple n o des. We design a framework named MimoSketch for the MIMO-specific scenario, which improves the fundamental mining tasks of item frequency estimation, item size distribution estimation, heavy hitter detection, heavy change detection, and entropy estimation. MimoSketch consists of an algorithm design and a policy to schedule items to nodes. MimoSketch's algorithm applies random counting to preserve a mathematically proven unbiasedness property, which makes it friendly to the aggregate query on multiple nodes; its memory layout is dynamically adaptive to the runtime item size distribution, which maximizes the estimation accuracy by storing more items. MimoSketch's scheduling policy balances items among nodes, avoiding nodes being overloaded or underloaded, which improves the overall mining accuracy. Our prototype and evaluation show that our algorithm can improve the accuracy of five typical mining tasks by an order of magnitude compared with the state-of-the-art solutions, and the scheduling policy further promotes the performance in MIMO scenarios.