Communication is pivotal in distributed training and a thorough understanding of its characteristics is essential for future optimizations. However, prior works are limited, either focusing on customized optimizations or conducting incomplete explorations on communication characteristics. In this work, we systematically analyze the communication characteristics of distributed training, considering two key aspects of communication: pattern and overhead, and assessing a broad spectrum of determinant factors. In particular, we extensively investigate the features of communication patterns, such as predictability, and comprehensively evaluate the impact of various factors on communication overhead. Additionally, we develop and validate an analytical formulation to estimate communication overhead, providing a mathematical understanding of models with predictability.
Recently, researchers have proposed ML-driven traffic engineering (TE) schemes where a neural network model is used to produce TE decisions in lieu of conventional optimization solvers. Unfortunately existing ML-based TE schemes are not explicitly designed to be robust to topology changes that may occur due to WAN evolution, failures or planned maintenance. In this paper, we present HARP, a neural model for TE explicitly capable of handling variations in topology including those not observed in training. HARP is designed with two principles in mind: (i) ensure invariances to natural input transformations (e.g., permutations of node ids, tunnel reordering); and (ii) align neural architecture to the optimization model. Evaluations on a multi-week dataset of a large private WAN show HARP achieves an MLU at most 11% higher than optimal over 98% of the time despite encountering significantly different topologies in testing relative to training data. Further, comparisons with state-of-the-art ML-based TE schemes indicate the importance of the mechanisms introduced by HARP to handle topology variability. Finally, when predicted traffic matrices are provided, HARP outperforms classic optimization solvers achieving a median reduction in MLU of 5 to 10% on the true traffic matrix.
26 September 2018 in IEEE Journal on Selected Areas in Communications
To optimize network cost, performance, and reliability, SDN advocates for centralized traffic engineering (TE) that enables more efficient path and egress point selection as well as bandwidth allocation. In this work, we argue SDN-based TE for ISP networks can be very challenging. First, ISP networks often are very large in size, imposing significant scalability challenges to the centralized TE. Second, ISP networks usually have diverse types of links, switches, and cost models, leading to a complex combination of optimizations. Third, ISP networks not only have many choices of internal paths but also include rich selections of egress points and interdomain routes, unlike cloud/enterprise networks. To overcome these challenges, we present a novel TE application framework, called Dragon, for existing SDN control planes. To address the scalability challenge, Dragon consists of hierarchical and recursive TE algorithms and mechanisms that divide flow optimization problems into subtasks and execute them in parallel. Further, Dragon allows ISPs to express diverse objectives for different parts of their network. Finally, we extend Dragon to jointly optimize the selection of intradomain and interdomain paths. Using extensive evaluation on real topologies and prototyping with SDN controller and switches, we demonstrate that Dragon outperforms existing TE methods both in speed and in optimality.
Routing dynamics heavily influence Internet data plane performance. Existing studies only narrowly focused on a few destinations and did not consider the predictability of the impact of routing changes on performance metrics such as reachability. In this work, we propose an efficient framework to capture coarse-grained but important performance degradation as a result of BGP routing events using light-weight probing. We deployed our framework across six vantage points for 11 weeks and found that the data plane experienced serious performance degradation in the form of reachability loss and forwarding loops following a significant fraction of updates affecting many destination prefixes and networks across all vantage points studied. Specifically, more than 39% of updates resulted in reachability loss, some lasting for more than 300 seconds, impacting more than 72% of probed prefixes and more than 35% of all the prefixes on the Internet. We identified that more than half of the prefixes have predictable routing behavior. Based on the stationarity of the correlation between routing changes and the data plane performance, we developed a model to accurately predict the severity of the impact due to routing changes. Such a model is directly helpful for making informed decisions for improved routing schemes such as overlay routing and backup path selection.