To ensure seamless, programmatic access to data for High Performance Computing (HPC) and analysis across multiple research domains, it is vital to have a methodology for standardization of both data and services. At the Australian National Computational Infrastructure (NCI) we have developed a Data Quality Strategy (DQS) that currently provides processes for: (1) Consistency of data structures needed for a High Performance Data (HPD) platform; (2) Quality Control (QC) through compliance with recognized community standards; (3) Benchmarking cases of operational performance tests; and (4) Quality Assurance (QA) of data through demonstrated functionality and performance across common platforms, tools and services. By implementing the NCI DQS, we have seen progressive improvement in the quality and usefulness of the datasets across the different subject domains, and demonstrated the ease by which modern programmatic methods can be used to access the data, either in situ or via web services, and for uses ranging from traditional analysis methods through to emerging machine learning techniques. To help increase data re-usability by broader communities, particularly in high performance environments, the DQS is also used to identify the need for any extensions to the relevant international standards for interoperability and/or programmatic access.

We present an implementation of parallel I∕O in the Modular Ocean Model (MOM), a numerical ocean model used for climate forecasting, and determine its optimal performance over a range of tuning parameters. Our implementation uses the parallel API of the netCDF library, and we investigate the potential bottlenecks associated with the model configuration, netCDF implementation, the underpinning MPI-IO library/implementations and Lustre filesystem. We investigate the performance of a global 0.25^∘ resolution model using 240 and 960 CPUs. The best performance is observed when we limit the number of contiguous I∕O domains on each compute node and assign one MPI rank to aggregate and to write the data from each node, while ensuring that all nodes participate in writing this data to our Lustre filesystem. These best-performance configurations are then applied to a higher 0.1^∘ resolution global model using 720 and 1440 CPUs, where we observe even greater performance improvements. In all cases, the tuned parallel I∕O implementation achieves much faster write speeds relative to serial single-file I∕O, with write speeds up to 60 times faster at higher resolutions. Under the constraints outlined above, we observe that the performance scales as the number of compute nodes and I∕O aggregators are increased, ensuring the continued scalability of I∕O-intensive MOM5 model runs that will be used in our next-generation higher-resolution simulations.

In recent years, magnetotelluric (MT) processing has become computationally intensive as the scale and size of MT surveys being run increases. Consequently, High Performance Computing (HPC) is now becoming a valuable tool for timely processing and modelling of these large MT datasets. As part of the MT component of the 2017-2019 Australian Research Data Commons (ARDC) funded Geoscience Data Enhanced Virtual Laboratory (DeVL) continuity project, The National Computational Infrastructure (NCI) at the Australian National University will enable MT datasets from The University of Adelaide to be added to the NCI HPC platform with the goal of creating a more Findable, Accessible, Interoperable, Reusable (FAIR) and open public resource. A focus will be on making the time series datasets more suitable for use on HPC and more interoperable with other Earth science disciplines, where High Performance Data (HPD) formats will allow for better scalability and performance. Metadata attributes, as defined by the Australian MT research community, will be added directly to the time series data files. Additionally, time series processing and 3D inversion codes are being optimised for HPD/HPC, with the end goal of rapid time series processing and 3D inversion. Making FAIR MT time series available on HPC can lead to a transformative change in the way MT data analysis is routinely conducted and such a change has the capacity to create new ways of doing collaborative and transparent MT analysis.

We present an implementation of parallel I/O in the Modular Ocean Model (MOM), a numerical ocean model used for climate forecasting, and determine its optimal performance over a range of tuning parameters. Our implementation uses the parallel API of the netCDF library, and we investigate the potential bottlenecks associated with the model configuration, netCDF implementation, the underpinning MPI-IO library/implementations and Lustre filesystem. We investigate the performance of a global 0.25° resolution model using 240 and 960 CPUs. The best performance is observed when we limit the number of contiguous I/O domains on each compute node and assign one MPI rank to aggregate and write the data from each node, while ensuring that all nodes participate in writing this data to our Lustre filesystem. These best performance configurations are then applied to a higher 0.1° resolution global model using 720 and 1440 CPUs, where we observe even greater performance improvements. In all cases, the tuned parallel I/O implementation achieves much faster write speeds relative to serial single-file I/O, with write speeds up to 60 times faster at higher resolutions. Under the constraints outlined above, we observe that the performance scales as the number of compute nodes and I/O aggregators are increased, ensuring the continued scalability of I/O-intensive MOM5 model runs that will be used in our next generation higher resolution simulations.