Since its inception in 2007, the High Performance Computing (HPC) program at Clemson University has experienced significant growth. This paper aims to provide a comprehensive overview of the evolution and current capabilities of our HPC cluster, Palmetto, while also sharing the valuable lessons learned over 17 years of its administration. Specifically, we will discuss the application of these lessons to the design of our next-generation cluster, Palmetto 2, with a focus on enhancing data portability, improving networking, optimizing job scheduling, streamlining software management, and implementing more efficient user-support systems. It is our hope that the insights gained from our experiences will be beneficial to other cyberinfrastructure support groups seeking to modernize their HPC capabilities.
The Research Computing and Data team at Clemson University has been working to improve resource utilization on the Palmetto Cluster. We developed a tool we call jobperf to provide users an easy way to view resource usage of jobs. Originally developed for PBS, Slurm support was also added as the Palmetto Cluster transitioned schedulers. Jobperf provides both a CLI and web interface to view real-time, per-node CPU, GPU and memory usage statistics. The tool was developed to be portable, requiring no cluster-wide components and easily installable without administrator intervention. This was tested on five Slurm clusters available through ACCESS-CI in addition to Palmetto, and shown to work well on three of these external systems. We have released the tool to the community as open source and we hope this tool is useful to other institutions.
The development of administrative and computational research tools requires reliable programmatic interfaces with the cluster scheduler. The Research Computing and Data (RCD) team at Clemson University has developed the Palmetto API, a proxy for the native Slurm RESTful interface, slurmrestd, while providing advanced authentication, authorization, and caching. This paper details the design and implementation of this proxy, evaluates the performance benefits from caching, and verifies compatibility to existing slurmrestd clients. The result is a light-weight and secure method of exposing our cluster scheduler to tools and automations.