Daniel Araújo De Medeiros

Daniel Araújo De Medeiros

Auto-generated profile, based on publicly available information

Daniel Araújo De Medeiros

KTH Royal Institute of Technology, Sweden

Research Focus & Metrics

Academic Career Length

4Years (2023-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
27 February 2023
Verified
Conference Paper
Open access
DMDaniel Araújo De Medeiros
Daniel Araújo De Medeiros
Stefano MarkidisStefano Markidis
Stefano Markidis
IPIvy Bo Peng
Ivy Bo Peng
Recently, federated HPC and cloud resources are becoming increasingly strategic for providing diversified and geographically available computing resources. However, accessing data stores across HPC and cloud storage systems is challenging. Many cloud providers use object storage systems to support their clients in storing and retrieving data over the internet. One popular method is REST APIs atop the HTTP protocol, with Amazon’s S3 APIs being supported by most vendors. In contrast, HPC systems are contained within their networks and tend to use parallel file systems with POSIX-like interfaces. This work addresses the challenge of diverse data stores on HPC and cloud systems by providing native object storage support through the unified MPI I/O interface in HPC applications. In particular, we provide a prototype library called LibCOS that transparently enables MPI applications running on HPC systems to access object storage on remote cloud systems. We evaluated LibCOS on a Ceph object storage system and a traditional HPC system. In addition, we conducted performance characterization of core S3 operations that enable individual and collective MPI I/O. Our evaluation in HACC, IOR, and BigSort shows that enabling diverse data stores on HPC and Cloud storage is feasible and can be transparently achieved through the widely adopted MPI I/O. Also, we show that a native object storage system like Ceph could improve the scalability of I/O operations in parallel applications.

7 May 2024
Verified
Conference Paper
Open access
DMDaniel Medeiros
Daniel Medeiros
EGEric B. Gregory
Eric B. Gregory
PCPhilippe Couvee
Philippe Couvee
JHJames Hawkes
James Hawkes
SGSebastien Gougeaud
Sebastien Gougeaud
PDPhilippe Deniel
Philippe Deniel
The new emerging scientific workloads to be executed in the upcoming exascale supercomputers face major challenges in terms of storage, given their extreme volume of data. In particular, intelligent data placement, instrumentation, and workflow handling are central to application performance. The IO-SEA project developed multiple solutions to aid the scientific community in adressing these challenges: a Workflow Manager, a hierarchical storage management system, and a semantic API for storage. All of these major products incorporate additional minor products that support their mission. In this paper, we discuss both the roles of all these products and how they can assist the scientific community in achieving exascale performance.