Reflections on Building Portable Benchmarking Frameworks

ICS 2026 Workshop on Reproducible and Sustainable HPC Benchmarking

Tuomas Koskela

UCL ARC

2026-07-06

Who am I?

  • I am a research software engineer at UCL Advanced Research Computing Centra (ARC)
  • I lead a team of HPC Research Software Engineer
    • We develop HPC research applications and tools
    • We work on research grants with researchers
    • We usually are not the main developers of the applications
  • I don’t run an HPC system

Motivation

We all Want Automatic Benchmarks

  • Developers
  • Researchers
  • Sysadmins
  • Procurements

Ingredients for Automated Benchmarking

  • Figures of Merit
  • Wrapping around build systems
  • Templating for scheduler scripts
  • Output parsing
  • Logging

Disclosure: I did this too

Excalibur-tests project

  • Driven by ExCALIBUR H&ES and research at Bristol and Met Office
  • https://ukri-excalibur.github.io/excalibur-tests/
  • Glue code for Spack and ReFrame
  • Clear abstraction of platforms and benchmarks
  • Support wide range of platforms and benchmarks
  • Engage with local admins to maintain environments

T Koskela et. al. 2023. Principles for Automated and Reproducible Benchmarking. In Proceedings of the SC ’23 Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis.

Principles for Automated and Reproducible Benchmarking

  1. A benchmark application should measure the efficiency of the application on a given platform.
  2. Teach the build system to build the benchmark using the best known parameters on each platform.
  3. Rebuild the benchmark every time it runs to guarantee the steps to reproduce the binary are known.
  4. Capture all steps taken to build the benchmark.
  5. Capture all steps to run the benchmark.
  6. Collect and post-process the data in a programmable way

T Koskela et. al. 2023. Principles for Automated and Reproducible Benchmarking. In Proceedings of the SC ’23 Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis.

Feedback from the UK Community

  • We gave a number of talks, workshops and hackthons about benchmarking
  • Received positive feedback from workshop attendees and collaborators
  • Users:
    • 1 team of RSE’s (UCL)
    • 1 research group (Bristol)
    • 1 team of HPC admins (DiRAC)

What are we Doing Now?

UKRI Living Benchmarks Project

  • “Funding mechanisms in the UK can make good benchmarking for procurement hard”
  • Provide a curated set of HPC benchmarks for UK acedemia and industry
  • Open source and publicly available at https://github.com/ukri-bench
  • Supporting the UK Next National Supercomputing Service

Other Use Cases

  • DiRAC
    • HPC facility for UK theoretical physics community
    • Early users of excalibur-tests, currently not maintained
  • Square Kilometre Array UK Regional Centre

Reflections

  • Portability is hard and takes continuous maintenance
    • Funding software maintenance is challenging
  • Institutional and individual buy-in are both important
    • Portability across institutions is also challengin
  • The tools you use don’t matter, as long they enable reproducibility

Thank you for listening!