Next-generation systems face a wide range of new potential sources of application interference, including resilience actions, system software adaptation, and in situ analytics programs. In this paper, we present a new model for analyzing the performance of bulk-synchronous HPC applications based on the use of extreme value theory. After validating this model against both synthetic and real applications, the paper then uses both simulation and modeling techniques to profile next-generation interference sources and characterize their behavior and performance impact on a selection of HPC benchmarks, mini-applications, and applications. Lastly, this work shows how the model can be used to understand how current interference mitigation techniques in multi-processors work.

Cities worldwide use camera systems that collect and store large amounts of images, which are used to study vehicle traffic conditions, facilitating traffic management authorities’ decision-making. Typically, the inspection of those images is performed manually, which prevents extracting relevant information in a timely manner. There is a lack of platforms to collect and analyze key data from traffic videos in an automatic and speedy way. Computer vision can be used in combination with parallel distributed systems to provide city authorities tools for automatic and fast processing of stored videos to determine the most significant driving patterns that cause traffic accidents while allowing to measure the traffic density. We use a Convolutional Neural Network (CNN) to detect vehicles captured by traffic cameras, which are then tracked using an algorithm that we designed, based on multi-tracking Kalman filters. To speed up analysis, we propose a low-cost distributed infrastructure based on Hadoop and Spark frameworks for data processing: videos are equally divided and distributed to multicore CPU nodes for analysis. However, splitting up videos could generate inaccuracies in vehicle counting, which were avoided through the use of an algorithm that we present in this work. We found that it is possible to rapidly determine traffic densities, identify dangerous driving maneuvers, and detect accidents with high accuracy by using low-cost commodity cluster computing. There is a lack of computing platforms to collect and analyze key data from traffic videos in an automatic and speedy way. Computer vision can be used in combination with parallel distributed systems to provide city authorities tools for automatic and fast processing of stored videos to determine the most significant driving patterns that cause traffic accidents while allowing to measure the traffic density. This study explores the integration of different tools such as parallel data processing, deep learning, and probabilistic models. We present an approach based on Convolutional Neural Network (CNN) and Kalman filters to detect and track vehicles captured by traffic cameras. To speed up analysis, we propose and evaluate a low-cost distributed infrastructure based on Hadoop and Spark frameworks and comprised of multicore CPU nodes for data processing. Finally, we present an algorithm to allow vehicle counting while avoiding inaccuracies generated when videos are split to be distributed for analysis. We found that it is possible to rapidly determine traffic densities, identify dangerous driving maneuvers, and detect accidents with high accuracy by using low-cost commodity cluster computing.

Fault tolerance and the availability of applications, computing infrastructure, and communications systems during unexpected events are critical in cloud environments. The microservices architecture, and the technologies that it uses, should be able to maintain acceptable service levels in the face of adverse circumstances. In this paper, we discuss the challenges faced by cloud infrastructure in relation to providing resilience to applications. Based on this analysis, we present our approach for a software platform based on a microservices architecture, as well as the resilience mechanisms to mitigate the impact of infrastructure failures on the availability of applications. We demonstrate the capacity of our platform to provide resilience to analytics applications, minimizing service interruptions and keeping acceptable response times.

The move towards high-performance computing (HPC) applications comprised of coupled codes and the need to dramatically reduce data movement is leading to a reexamination of time-sharing vs. space-sharing in HPC systems. In this paper, we discuss and begin to quantify the performance impact of a move away from strict space-sharing of nodes for HPC applications. Specifically, we examine the potential performance cost of time-sharing nodes between application components, we determine whether a simple coordinated scheduling mechanism can address these problems, and we research how suitable simple constraint-based optimization techniques are for solving scheduling challenges in this regime. Our results demonstrate that current general-purpose HPC system software scheduling and resource allocation systems are subject to significant performance deficiencies which we quantify for six representative applications. Based on these results, we discuss areas in which additional research is needed to meet the scheduling challenges of next-generation HPC systems.

Summary: We present a detailed examination of time agreement characteristics for nodes within extreme‐scale parallel computers. Using a software tool we introduce in this paper, we quantify attributes of clock skew among nodes in three representative high‐performance computers sited at three national laboratories. Our measurements detail the statistical properties of time agreement among nodes and how time agreement drifts over typical application execution durations. We discuss the implications of our measurements, why the current state of the field is inadequate, and propose strategies to address observed shortcomings.

Next-generation applications increasingly rely on in situ analytics to guide computation, reduce the amount of I/O performed, and perform other important tasks. Scheduling where and when to run analytics is challenging, however. This paper quantifies the costs and benefits of different approaches to scheduling applications and analytics on nodes in large-scale applications, including space sharing, uncoordinated time sharing, and gang scheduled time sharing.