As part of the Development Operations (DevOps) School for HPC, an integral, independent educational program during the CARLA conference, Dr. Christian Iwainsky, our expert from Darmstadt, gave a presentation at the end of September on “Boosting HPC Systems with Job-Specific Monitoring and Automated Inefficiency Detection”.

More about the talk

High-performance computing (HPC) systems are critical enablers of modern scientific research. However, their efficient operation requires more than raw performance – it also depends on a deeper understanding by users of how applications behave and how resources can be used efficiently. While traditional system monitoring focuses on infrastructure-wide health, it often misses the job-level context that determines whether hardware is being used effectively.

Christian Iwainsky explains: "In my presentation, I explored how scalable, job-specific performance monitoring can fill this gap. Based on our operational experiences at TU Darmstadt, I presented how ClusterCockpit provides DevOps-oriented access to actionable HPC performance data." It enables low-overhead, job-level monitoring of key metrics such as CPU load, memory bandwidth, I/O activity, and energy consumption. These data help identify inefficiencies and support data-driven optimization in HPC environments.

"Building on this foundation, the PathoJobs system - a project I have been closely involved with - applies rule-based classification to automatically detect inefficiencies in large volumes of job execution data. This provides timely, interpretable feedback to users and support teams and facilitates targeted consulting and tuning," Iwainsky continues.

By integrating job-centric observability and automated analysis into daily operations, HPC centers can build a solid foundation for supporting both scientific outcomes and efficient system usage - even under tight resource constraints. The talk concluded with lessons learned from production deployments and a looked at current research and emerging trends in adaptive, job-aware monitoring.

About CARLA (Dates: September, Monday 22 – Friday 26, 2025)

CARLA is an international conference aimed at providing a forum to foster the growth and strength of the High Performance Computing (HPC) community in Latin America and beyond. The conference serves as a platform for new ideas, techniques, and research in HPC and its application areas, and it started in 2014. This year, CARLA took place in the Caribbean for the first time - in Kingston, Jamaica.

CARLA has become the flagship conference for HPC in the region and invites the international community to share its advances on both HPC and HPC4AI, as these fields are key areas that are becoming the predominant engine for innovation and development.

Development Operations (DevOps) School for HPC

The Development Operations (DevOps) School for HPC is an integral, standalone educational offering during the CARLA 2025 conference. The School is tailored to equip participants with hands-on skills and conceptual insights in managing and optimizing HPC systems. Led by renowned instructors, attendees can expect high-quality instruction and international expertise to equip participants with hands-on skills and conceptual insights in managing and optimizing large clusters and HPC systems.

The program is rich in content and practical engagement—offering invited keynotes, deep-dives on tools like Ansible, Spack, EasyBuild, and XDMoD, as well as hands-on tutorials and sessions on benchmarking, monitoring, and profiling.