fix continuous batching issues, extend ut cases to xpu #41830
Conversation
Signed-off-by: Yao, Matrix <matrix.yao@intel.com>
yao-matrix
changed the title
|
@SunMarc , could you pls take a review? Thx very much. |
remi-or
left a comment
remi-or
left a comment
There was a problem hiding this comment.
LGTM! Thanks for adding, I just have 2 nits :)
| Expectations, | ||
| require_kernels, | ||
| require_torch_accelerator, | ||
| require_torch_gpu, |
There was a problem hiding this comment.
Is @require_torch_gpu still used after those changes?
There was a problem hiding this comment.
@remi-or paged attention enabling on XPU in kernels is still ongoing, suppose be ready soon, then we will transfer the left cases from CUDA-only to XPU, at this time, we still have some paged attention cases which are still CUDA only.
| blocks = blocks.reshape(local_experts, -1, module.intermediate_size // 2) | ||
| if getattr(target_device, "type", target_device) == "cpu": | ||
| target_device = "cuda" | ||
| target_device = torch.accelerator.current_accelerator().type if hasattr(torch, "accelerator") else "cuda" |
There was a problem hiding this comment.
if @SunMarc can OK this, not familiar w/ different accelerators
|
@SunMarc , I think we are OK to go now, thx very much. |
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
…1830) * extend conrinuous batching cases to xpu Signed-off-by: Yao, Matrix <matrix.yao@intel.com> * fix style Signed-off-by: Yao, Matrix <matrix.yao@intel.com> * fix style Signed-off-by: Yao, Matrix <matrix.yao@intel.com> --------- Signed-off-by: Yao, Matrix <matrix.yao@intel.com> Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com>
4 participants
@SunMarc , pls help review, thx very much.