In this paper we propose a distributed architecture to provide machine learning practitioners with a set of tools and cloud services that cover the whole machine learning development cycle: ranging from the models creation, training, validation and testing to the models serving as a service, sharing and publication. In such respect, the DEEP-Hybrid-DataCloud framework allows transparent access to existing e-Infrastructures, effectively exploiting distributed resources for the most compute-intensive tasks coming from the machine learning development cycle. Moreover, it provides scientists with a set of Cloud-oriented services to make their models publicly available, by adopting a serverless architecture and a DevOps approach, allowing an easy share, publish and deploy of the developed models.

The Sentinel-1 satellite system continuously observes European countries at a relatively high revisit frequency of six days per orbital track. Given the Sentinel-1 configuration, most areas in Czechia are observed every 1–2 days by different tracks in a moderate resolution. This is attractive for various types of analyses by various research groups. The starting point for interferometric (InSAR) processing is an original data provided in a Single Look Complex (SLC) level. This work represents advantages of storing data augmented to a specifically corrected level of data, SLC-C. The presented database contains Czech nationwide Sentinel-1 data stored in burst units that have been pre-processed to the state of a consistent well-coregistered dataset of SLC-C. These are resampled SLC data with their phase values reduced by a topographic phase signature, ready for fast interferometric analyses (an interferogram is generated by a complex conjugate between two stored SLC-C files). The data can be used directly into multitemporal interferometry techniques, e.g., Persistent Scatterers (PS) or Small Baseline (SB) techniques applied here. A further development of the nationwide system utilising SLC-C data would lead into a dynamic state where every new pre-processed burst triggers a processing update to detect unexpected changes from InSAR time series and therefore provides a signal for early warning against a potential dangerous displacement, e.g., a landslide, instability of an engineering structure or a formation of a sinkhole. An update of the processing chain would also allow use of cross-polarised Sentinel-1 data, needed for polarimetric analyses. The current system is running at a national supercomputing centre IT4Innovations in interconnection to the Czech Copernicus Collaborative Ground Segment (CESNET), providing fast on-demand InSAR results over Czech territories. A full nationwide PS processing using data over Czechia was performed in 2017, discovering several areas of land deformation. Its downsampled version and basic findings are demonstrated within the article.

An adequate compute and storage infrastructure supporting the full exploitation of Copernicus and Earth Observation datasets is currently not available in Europe. This paper presents the cross-disciplinary open-source technologies being leveraged in the C-SCALE project to develop an open federation of compute and data providers as an alternative to monolithic infrastructures for processing and analysing Copernicus and Earth Observation data. Three critical aspects of the federation and the chosen technologies are elaborated upon: (1) federated data discovery, (2) federated access and (3) software distribution. With these technologies the open federation aims to provide homogenous access to resources, thereby enabling its users to generate meaningful results quickly and easily. This will be achieved by abstracting the complexity of infrastructure resource access provisioning and orchestration, including discovery of data across distributed archives, away from the end-users. Which is needed because end-users wish to focus on analysing ready-to-use data products and models rather than spending their time on the setup and maintenance of complex and heterogeneous IT infrastructures. The open federation will support processing and analysing the vast amounts of Copernicus and Earth Observation data that are critical for the implementation of the Destination Earth resp. Digital Twins vision for a high precision digital model of the Earth to model, monitor and simulate natural phenomena and related human activities.

One of the benefits of OCCI stems from simplifying the life of developers aiming to integrate multiple cloud managers. It provides them with a single protocol to abstract the differences between cloud service implementations used on sites run by different providers. This comes particularly handy in federated clouds, such as the EGI Federated Cloud Platform, which bring together providers who run different cloud management platforms on their sites: most notably OpenNebula, OpenStack, or Synnefo. Thanks to the wealth of approaches and tools now available to developers of virtual resource management solutions, different paths may be chosen, ranging from a small-scale use of an existing command line client or single-user graphical interface, to libraries ready for integration with large workload management frameworks and job submission portals relied on by large science communities across Europe. From lone wolves in the long-tail of science to virtual organizations counting thousands of users, OCCI simplifies their life through standardization, unification, and simplification. Hence cloud applications based on OCCI can focus on user specifications, saving cost and reaching a robust development life-cycle. To demonstrate this, the paper shows several EGI Federated Cloud experiences, demonstrating the possible approaches and design principles.

This paper provides a brief introduction into emerging user requirements and expectations in the context of large cloud-based infrastructures. Observed cloud usage patterns are used to demonstrate the need for large federated cloud infrastructures, in particular cloud federations bridging the gap between private cloud infrastructures and large commercial public cloud providers. The paper proposes the use of open standards as the best possible way to achieve a working user-friendly large-scale cloud federation and discusses the challenges in assembling and managing such a federation whilst focusing on the differences between private-private and public-private cloud federations in key areas of virtual machine management and authentication, authorization, and identity management.