The Case of the Elusive Application Performance
on Production GPU Supercomputers

Abstract

Modern HPC facilities increasingly rely on GPU-accelerated clusters to drive both scientific computing and AI workloads. Performance variability is a critical issue in these systems, undermining efficiency and performance reproducibility. While prior studies have extensively analyzed variability in CPU-centric supercomputers, similar large-scale investigations on GPU clusters are lacking.

To address this gap, we set up a longitudinal experiment on two state-of-the-art GPU-based supercomputers: NERSC's Perlmutter and ORNL's Frontier. We benchmark several representative HPC and AI applications and collect detailed performance data including network counters, profiling output, and job scheduler logs. We analyze this data to identify the impact of compute performance variations, allocated node topology, and network conditions on the overall runtime variability. We also use a machine learning based approach to identify potential correlations between these factors, and to forecast performance variability. We provide actionable insights for both system administrators and users to mitigate or predict performance variations in GPU-accelerated HPC environments.

Study Design

Longitudinal experiments on two flagship GPU supercomputers over ~4 months in 2024–2025

Applications Studied

Application Breakdown Analysis

Execution time dissected into compute vs. communication components, from fastest to slowest observed runs. Communication time — not compute — drives variability across all applications.

HPC Applications: AMG2023 & MILC

AMG2023 — Perlmutter: ~74% of runtime in MPI; slowest runs have ~40% more communication

AMG2023 — Frontier: ~84% of runtime in MPI; improvement visible post Jan 14 patch

MILC — Perlmutter: ~56% in MPI; Allreduce and Test are the most variable routines

MILC — Frontier: slowest communication up to 122% higher than fastest runs

AI Applications: DeepCAM & nanoGPT

DeepCAM — Perlmutter: Allreduce up to 4× slower in worst runs vs. best

DeepCAM — Frontier: Allreduce up to 24× slower in worst vs. best; compute is stable

nanoGPT — Perlmutter: Allreduce 3× slower in worst vs. best; Allgather also variable

nanoGPT — Frontier: 4D sub-communicator design reduces Allreduce variability significantly

Allreduce Distribution — Long Tails on Perlmutter

Allreduce is the single most variable routine across all applications. These violin plots show the full distribution of MPI and NCCL Allreduce times — including the long-tail outlier runs that can be dramatically slower than typical.

MPI Allreduce distribution across AMG2023 and MILC on Perlmutter — note long upper tails

NCCL Allreduce distribution across DeepCAM and nanoGPT on Perlmutter

Impact of Job Placement

Does allocating nodes across more dragonfly groups (fragmented placement) hurt performance? The answer is mostly no — temporal network conditions matter far more than topology.

nanoGPT on Perlmutter: number of dragonfly groups assigned to the job vs. runtime. No strong trend — placement fragmentation does not significantly hurt performance.

DeepCAM on Frontier: same analysis. High variance exists within any given number of groups, indicating other factors (network congestion) dominate.

ML-Based Performance Prediction

XGBoost regression trained on system metrics accurately predicts runtime variability

Step 1 — NIC Counter Correlation with Runtime

Spearman's correlation coefficients between application execution times and NIC counter values (mean and max). Positive = higher counter values → slower runtime. Key counters: rh:sct_timeouts (retries from packet loss), hni_rx_paused_0 (receive path backpressure).

Spearman correlation heatmap between application runtime and NIC counter values on Perlmutter. While individual counters show partial signal, multi-counter ML models are needed to capture the full complexity of variability.

Step 2 — XGBoost Feature Importances

After training the XGBoost model on placement, GEMM, Allreduce, and NIC counter features, we examine which features matter most for runtime prediction on each system.

Perlmutter (left): hni_rx_paused_0_mean and NCCL Allreduce (2GB) dominate — NIC receive-path stalls and network-wide congestion are the key signals.
Frontier (right): lpe_net_match_request_0, atu_cache_hit_derivative1, and parbs_tarb_pi_non_posted_blocked_cnt dominate — local data movement bottlenecks drive Frontier variability.

Step 3 — Prediction Accuracy (MAPE & Direction Accuracy)

We evaluate prediction quality with two metrics: MAPE (lower is better) and Direction Accuracy (DA ≈ 1.0 means the model correctly identifies when performance is improving vs. degrading). Results shown for incremental feature subsets.

MAPE and DA for all four applications on both systems, across four feature combinations: Placement only → + GEMM → + Allreduce → + NIC Counters (full). Adding NIC counters yields the largest accuracy gains, especially for high-variability apps like DeepCAM. Without NIC counters, DA can fall near chance level; with them it approaches 1.0.

Step 4 — Predicted vs. Actual Runtime

Scatter plots of predicted vs. actual runtimes using the full feature set (placement + GEMM + Allreduce + NIC counters). Points close to the diagonal indicate accurate predictions.

Perlmutter: predicted vs. actual runtime for all four applications. The model tracks absolute performance well for most runs; some deviations arise from lack of router (Rosetta) counters.

Frontier: predicted vs. actual runtime. Model generalizes well even for MILC with only 7 training samples — demonstrating cross-application transferability.

BibTeX

@inproceedings{wei2026elusiveperformance,
  title     = {The Case of the Elusive Application Performance on
               Production {GPU} Supercomputers},
  author    = {Wei, Cunyang and Pradeep, Keshav and Bhatele, Abhinav},
  booktitle = {Proceedings of the IEEE International Parallel and
               Distributed Processing Symposium (IPDPS)},
  year      = {2026},
}