`profiler + cudagraph` hangs forever for rope

[BoyuanFeng]

profiler + cudagraph hangs forever for rope on main branch.

repro: python3 run.py --op rope --metrics=latency --latency-measure-mode=profiler --cudagraph

[BoyuanFeng]

There still seems to be gaps between --cudagraph and --latency-measure-mode=profiler

[xuzhao9]

--metrics=latency --cudagraph uses do_bench_cudagraph and I believe it is less accurate than --latency-measure-mode=profiler --metrics=latency because the former does not clear L2 cache, and maybe because of that, its latency measure is more unstable than using --latency-measure-mode=profiler.

As for the GPU hang issue, can you please create a minimal reproduction script? Something like:

rope, input = get_kernel_and_input()
profiler_config = {
        "activities": [
            torch.profiler.ProfilerActivity.CPU,
            torch.profiler.ProfilerActivity.CUDA,
        ],
        "record_shapes": False,
        "profile_memory": False,
        "with_stack": False,
}
fn = lambda: rope(input)

g = torch.cuda.CUDAGraph()
with torch.cuda.graph(g):
    fn()

with torch.profiler.profile(**profiler_config) as prof:
    g.replay()
torch.cuda.synchronize()

I want to further triage this problem whether it is because of the benchmarking harness, or it is from the kernel itself.

[davidberard98]

which cuda version are you building with @BoyuanFeng ? CUDA < 12.6 is known to have issues with cudagraphs + cupti sometimes hanging

[BoyuanFeng]

@davidberard98 i'm using cuda 12.9

[BoyuanFeng]

--metrics=latency --cudagraph uses do_bench_cudagraph and I believe it is less accurate than --latency-measure-mode=profiler --metrics=latency because the former does not clear L2 cache, and maybe because of that, its latency measure is more unstable than using --latency-measure-mode=profiler.

since do_bench_cudagraph does not clear L2 cache, I expect it to be faster than --latency-measure-mode=profiler --metrics=latency? But for (8192, 4096) shape, do_bench_cudagraph gives 0.066 vs 0.060 from --latency-measure-mode=profiler --metrics=latency. This is quite counter-intuitive.

[xuzhao9]

I tried to reproduce it on H100.

$ python3 run.py --op rope --metrics=latency --cudagraph --num-inputs 1 --input-id 2 
      (H, T)    apply_rotary_pos_emb-latency    liger_rotary_pos_emb-latency    torch_compile_rotary_pos_emb_full_op-latency
------------  ------------------------------  ------------------------------  ----------------------------------------------
(8192, 4096)               0.870288 (±0.06%)               0.333869 (±0.05%)                               0.176651 (±0.04%)

$ python3 run.py --op rope --metrics=latency --latency-measure-mode=profiler --num-inputs 1 --input-id 2 
      (H, T)    apply_rotary_pos_emb-latency    liger_rotary_pos_emb-latency    torch_compile_rotary_pos_emb_full_op-latency
------------  ------------------------------  ------------------------------  ----------------------------------------------
(8192, 4096)               0.878410 (±0.02%)               0.332768 (±0.17%)                               0.177401 (±0.04%)
     average

$ python3 run.py --op rope --metrics=latency --latency-measure-mode=profiler --num-inputs 1 --input-id 2  --cudagraph
      (H, T)    apply_rotary_pos_emb-latency    liger_rotary_pos_emb-latency    torch_compile_rotary_pos_emb_full_op-latency
------------  ------------------------------  ------------------------------  ----------------------------------------------
(8192, 4096)               0.870811 (±0.02%)               0.331132 (±0.06%)                               0.174895 (±0.05%)
     average

I also collected the traces:

1. kineto+cudagraph+L2 cache flush: https://www.internalfb.com/intern/perfdoctor/trace_view?filepath=tree/traces/tritonbench/devgpu089.cco2.facebook.com_3704523.1757601449790784676.pt.trace.json&bucket=pyper_traces, kernel latency 0.144 + 0.036 = 0.180 ms
2. kineto+L2 cache flush: https://www.internalfb.com/intern/perfdoctor/trace_view?filepath=tree/traces/tritonbench/devgpu089.cco2.facebook.com_3016797.1757600387046740095.pt.trace.json&bucket=pyper_traces, kernel latency 0.141+0.033 = 0.174 ms

From the traces it looks like to me that the latency variance is from either noise or inherent perf unstableness of the kernels. (However, this is contradict with the fact that variance level is 0.04%) I will rerun it a few more times to investigate.

[BoyuanFeng]

@xuzhao9 could you try without --num-inputs 1 --input-id 2? The hang issue repros when there are multiple inputs.
python3 run.py --op rope --metrics=latency --latency-measure-mode=profiler --cudagraph