How does Tritonbench use multi-GPU systems?

[bdbowyer]

More of a question than an issue. I'm not clear from the code if special setup is needed for Multi-GPU systems. If I'm launching to an 8 GPU system, should I use 8 Dockers each locked to one GPU, or will Tritonbench distribute the work to the 8 GPUS itself?

[xuzhao9]

Currently Tritonbench only contains single-GPU workloads, so I will suggest to use 8 Dockers each locked to one GPU.

Multi-GPU support is not on the roadmap yet, as we currently do not see a strong demand.
That being said, multi-GPU support can be added from two directions:

1. add multi-GPU workloads such as NVSHEMEM or NCCL tests
2. add multi-GPU benchmarking support to benchmark things in parallel for benchmarking efficiency.