With the increase of complex scientific simulations driven by workflows and heterogeneous workload profiles, managing system resources effectively is essential for improving performance and system throughput, especially due to trends like heterogeneous HPC and deeply integrated systems with on-chip accelerators. For optimal resource utilization, dynamic resource allocation can improve productivity across all system and application levels, by adapting the applications' configurations to the system's resources. In this context, malleable jobs, which can change resources at runtime, can increase the system throughput and resource utilization while bringing various advantages for HPC users (e.g., shorter waiting time). Malleability has received much attention recently, even though it has been an active research area for almost two decades [1]. This paper presents the state-of-the-art of malleable implementations in HPC systems, targeting mainly malleability in compute and I/O resources. Based on our experiences, we state our current concerns and list future opportunities for research.

This paper presents a novel feature for improving the scheduling process based on the performance prediction and the detection of CPU and I/O interference between applications. This feature consists of using malleable synthetic benchmarks – called clones – that reproduce the behaviour of applications executed in a cluster. These proxies can be used with two objectives: to build large and representative datasets that can be used to train the machine learning algorithms for modelling the platform workload, and to evaluate in advance if two executing applications can potentially produce contention hazards related to the shared use of the system resources like CPU, cache memory or I/O bandwidth. The proposed framework generates application clones based on generic-purpose performance information collected from monitoring. Unlike other works based on the use of micro-architectures or metrics obtained from simulators, in the approach presented in this work, the application clones generate similar behaviour without extracting data from the binaries, avoiding the necessity of managing code or data from the applications. One advantage of this approach is that they can be shared securely because they have not been generated using any piece of the original code. In addition, the proposed clones are malleable, so they also able to model the application behaviour under a different number of processes. In this work, we show how the use of clones contributes to improving the application scheduling by reducing the number of evaluations that is necessary to perform, and to detect performance degradation (interference) without the necessity of involving the execution of the actual applications. We evaluate the proposed clone generation approach on two sets of benchmarks (CPU and I/O oriented) and several applications. We also compare the performance obtained during the execution of the proxies and the applications to show their similarity. Finally, we include an evaluation of the interference detection using this novel approach.

This work presents a HPC framework that provides new strategies for resource management and job scheduling, based on executing different applications in shared compute nodes, maximizing platform utilization. The framework includes a scalable monitoring tool that is able to analyze the platform’s compute node utilization. We also introduce an extension of CLARISSE, a middleware for data-staging coordination and control on large-scale HPC platforms that uses the information provided by the monitor in combination with application-level analysis to detect performance degradation in the running applications. This degradation, caused by the fact that the applications share the compute nodes and may compete for their resources, is avoided by means of dynamic application migration. A description of the architecture, as well as a practical evaluation of the proposal, shows significant performance improvements up to 20% in the makespan and 10% in energy consumption compared to a non-optimized execution.