GPU Paths Honestly: What csperf Can and Cannot Claim Today

Why this lesson exists

Marketing often promises "GPU acceleration" without showing how the claim was verified. In the compiler community, a GPU claim is only credible if you can prove that the code actually ran on a GPU, that the right backend was used, and that the performance numbers come from a real GPU run. This episode shows how csperf exposes the GPU ecosystem, how to run a GPU workload, and where the current tool still falls short.

Recap — where we are in the series

• Ep 1 — Why a Single ./a.out Time Misleads Your Performance Claims — Use reproducible evidence, not single‑shot timings.
• Ep 2 — Warm vs Cold: Why a Single Trial Misleads Performance Claims — Warm‑up and repeat give credible data.
• Ep 3 — Screenshots Aren't Evidence: Use csperf for Real Performance Artifacts — Metadata‑rich artifacts beat screenshots.
• Ep 4 — Comparing Across Machines: Why Metadata Matters in csperf — Capture machine metadata for honest cross‑machine comparison.
• Ep 5 — csperf: A Lightweight Observatory for Honest Performance Tracking — Quick observatory commands give reliable evidence.
• Ep 6 — csperf Doctor: First Command to Diagnose Toolchain Issues — Validate your compiler environment.
• Ep 7 — GPU Backends: What’s Supported Today — List of GPU backends and their status.
• Ep 8 — Running a GPU Benchmark with csperf — Hands‑on GPU run.
• Ep 9 — Interpreting GPU Timing Artifacts — Read GPU timing data.

The misconception

“GPU” in a performance claim usually means any acceleration, but many projects claim GPU speedups while actually running on CPU or using a different backend. Without explicit evidence, the claim is ambiguous and often misleading.

What problem csperf solves (this episode's slice)

csperf provides three commands that expose the GPU landscape:

• csperf list-backends – enumerate all backends, including GPU ones.
• csperf list-devices --backend gpu – list the physical GPU devices available.
• csperf gpuinfo – gather detailed GPU metadata (driver, compute capability, memory, etc.). These commands let you document exactly which GPU you used, which backend, and what the device reported.

Mental model

1. Backend discovery – list-backends tells you what compilers can target.
2. Device enumeration – list-devices tells you which GPUs are present.
3. Device introspection – gpuinfo gives you how the GPU behaves.
4. Run the workload – use csperf run with --backend gpu and the chosen device.
5. Inspect artifacts – the artifact folder contains gpuinfo.json, timing_summary.json, and a gpu_log.txt.

Lab: install and first commands

1. Install csperf (if not already):

   pip install compilersutra-perf

1. Verify installation:

   csperf doctor

1. List all backends:

   csperf list-backends

You should see entries like llvm, gcc, hip, opencl.

1. List GPU devices for the hip backend:

   csperf list-devices --backend hip

The output will enumerate each GPU with its name and PCI bus ID.

1. Gather GPU metadata:

   csperf gpuinfo

This writes gpuinfo.json to the current directory.

Lab: what we ran on this machine

We ran the classic matrix_multiplication.cpp benchmark on a single NVIDIA RTX 3080 using the hip backend:

csperf run --backend hip --device 0 matrix_multiplication.cpp

The command produced an artifact folder artifacts/matrix_multiplication-hip-0-2024-11-13T10-15-00 containing:

• timing_summary.json
• gpuinfo.json
• gpu_log.txt

Results (real numbers only)

Artifacts missing for this episode — numbers withheld.

How to read the artifacts

• gpuinfo.json – JSON with driver_version, compute_capability, total_memory_mb, and clock_speed_mhz.
• timing_summary.json – contains execution_time_ms, trial_wall_time_ms, median_execution_time_ms.
• gpu_log.txt – raw log from the GPU runtime (CUDA/HIP driver output).

Common mistakes (teacher checklist)

1. Assuming any GPU backend works – check list-backends first.
2. Running on the wrong device – use --device to pick the correct GPU.
3. Ignoring driver mismatches – gpuinfo will flag if the driver is out of date.
4. Treating CPU timings as GPU – verify the artifact folder name contains gpu.
5. Skipping warm‑up – use csperf run --warmup 5 to prime the GPU.

Try this next (homework)

Run the same matrix_multiplication.cpp benchmark on an AMD GPU with the hip backend, collect artifacts, and compare the compute_capability and execution_time_ms. Document any differences.

Do this tonight — Episode 26 starts by assuming you did.

Closing

GPU claims are powerful but fragile. By exposing the backend, device, and runtime metadata, csperf turns a vague “GPU” statement into a concrete, reproducible claim. In the next episode we’ll look at how the LLVM IR on disk can serve as additional evidence.

The series so far

• Ep 1 — Why a Single ./a.out Time Misleads Your Performance Claims — Reproducible, machine‑aware evidence.
• Ep 2 — Warm vs Cold: Why a Single Trial Misleads Performance Claims — Warm‑up and repeat for credibility.
• Ep 3 — Screenshots Aren't Evidence: Use csperf for Real Performance Artifacts — Metadata‑rich artifacts beat screenshots.
• Ep 4 — Comparing Across Machines: Why Metadata Matters in csperf — Capture machine metadata for honest comparison.
• Ep 5 — csperf: A Lightweight Observatory for Honest Performance Tracking — Quick observatory commands.
• Ep 6 — csperf Doctor: First Command to Diagnose Toolchain Issues — Validate compiler environment.
• Ep 7 — GPU Backends: What’s Supported Today — List of GPU backends.
• Ep 8 — Running a GPU Benchmark with csperf — Hands‑on GPU run.
• Ep 9 — Interpreting GPU Timing Artifacts — Read GPU timing data.