19 August 2021 in IEEE Transactions on Parallel and Distributed Systems
Many extreme scale scientific applications have workloads comprised of a large number of individual high-performance tasks. The Pilot abstraction decouples workload specification, resource management, and task execution via job placeholders and late-binding. As such, suitable implementations of the Pilot abstraction can support the collective execution of large number of tasks on supercomputers. We introduce RADICAL-Pilot (RP) as a portable, modular and extensible pilot-enabled runtime system. We describe RP's design, architecture and implementation. We characterize its performance and show its ability to scalably execute workloads comprised of tens of thousands heterogeneous tasks on DOE and NSF leadership-class HPC platforms. Specifically, we investigate RP's weak/strong scaling with CPU/GPU, single/multi core, (non)MPI tasks and Python functions when using most of ORNL Summit and TACC Frontera. RADICAL-Pilot can be used stand-alone, as well as the runtime for third-party workflow systems.
18 December 2019 in Journal of Digital Science
Big Data is supposed to be one of the main traits of new coming digital era. Its technological aspects are usually widely discussed, whereas social peculiarities are mostly neglected. We present main approaches to Big Data, and argue that despite seeming revolutionary technology, Big Data can be treated as a new tool to produce knowledge. That means, it generates the same risks and challenges as other breakthroughs we witnessed previously. To our viewpoint, cultural aspects should be as counted as a main issue in Big Data implementation. Since the inability to control big data through prohibiting some peculiar features it possesses, we argue that one should focus on such practical steps as terminology improvements, and the evaluation of societal outcomes of the new technology.
24 December 2018 in Scientific Visualization