Intel ARC / XPU Support for Whisper ASR – An Open Invitation

[Sikerdebaard]

Hello Whisper ASR Team,

I've been having a blast using your web service. Inspired by your work, I decided to roll up my sleeves and add Intel ARC / XPU support. It's been quite the adventure, especially with the twists and turns of the Intel ARC ecosystem for deep learning.

Here's the catch though – the Docker image turned out to be a bit on the larger side, hitting the scales at about 25 GiB. So, I've held off on a pull request for now because that's quite a load to consider!

I'm reaching out to see if you're interested in this Intel ARC / XPU compatibility. If you are, my repo is ready and waiting to help as a starting point for weaving in support on your end. No pressure, though. Just thought I'd share what I've been up to and see if it could contribute to your toolkit.

If this sounds like something you'd want to explore, or if you have any insights, I'm here and happy to chat more.

My repo: https://github.com/Sikerdebaard/whisper-asr-webservice-xpu

[ayancey]

Please give us some time to review this. I don't have an ARC card, so testing could be difficult. Do you, @ahmetoner ?

[Oyo95]

Hi !

any news about the compatibility with Intel gpu arc ? :)

[ChezRD]

As far as I know, ctranslate2 has no support for XPU, so no faster-whisper too.
I used leuc/whisper on my ARC 750 with 8GB VRAM. This amount of VRAM can handle only medium model.

[bonswouar]

As far as I understand XPU are supported by huggingface transformers though, so it could be supported with #146

[Awlexus]

I've tried to get it running, but encountered a couple of problems on the way

1. Error with intel repository

3.247 W: GPG error: https://repositories.intel.com/gpu/ubuntu jammy InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 28DA432DAAC8BAEA
3.247 E: The repository 'https://repositories.intel.com/gpu/ubuntu jammy InRelease' is not signed.

I solved this by adding the following snippet taken from https://dgpu-docs.intel.com/driver/installation.html

RUN sudo apt-get install -y gpg-agent wget && \
		wget -qO - https://repositories.intel.com/gpu/intel-graphics.key | \
    sudo gpg --yes --dearmor --output /usr/share/keyrings/intel-graphics.gpg && \
		echo "deb [arch=amd64 signed-by=/usr/share/keyrings/intel-graphics.gpg] https://repositories.intel.com/gpu/ubuntu jammy/lts/2350 unified" | \
    sudo tee /etc/apt/sources.list.d/intel-gpu-jammy.list

2. Model does not seem to be using GPU
   This line here seems to tell me that it's using the CPU instead of the GPU. Looking at the CPU usage with top and the duration for transcribing anything confirms that theory. I have not found out how to fix that.

/usr/local/lib/python3.10/dist-packages/whisper/transcribe.py:117: UserWarning: FP16 is not supported on CPU; using FP32 instead

[rbrowning85]

Hello Whisper ASR Team,

I've been having a blast using your web service. Inspired by your work, I decided to roll up my sleeves and add Intel ARC / XPU support. It's been quite the adventure, especially with the twists and turns of the Intel ARC ecosystem for deep learning.

Here's the catch though – the Docker image turned out to be a bit on the larger side, hitting the scales at about 25 GiB. So, I've held off on a pull request for now because that's quite a load to consider!

I'm reaching out to see if you're interested in this Intel ARC / XPU compatibility. If you are, my repo is ready and waiting to help as a starting point for weaving in support on your end. No pressure, though. Just thought I'd share what I've been up to and see if it could contribute to your toolkit.

If this sounds like something you'd want to explore, or if you have any insights, I'm here and happy to chat more.

My repo: https://github.com/Sikerdebaard/whisper-asr-webservice-xpu

Any updates on this project? I just got an Intel Arc GPU and I'm very interested in running Whisper on it.

[rbrowning85]

I've tried to get it running, but encountered a couple of problems on the way

1. Error with intel repository

3.247 W: GPG error: https://repositories.intel.com/gpu/ubuntu jammy InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 28DA432DAAC8BAEA
3.247 E: The repository 'https://repositories.intel.com/gpu/ubuntu jammy InRelease' is not signed.

I solved this by adding the following snippet taken from https://dgpu-docs.intel.com/driver/installation.html

RUN sudo apt-get install -y gpg-agent wget &&
wget -qO - https://repositories.intel.com/gpu/intel-graphics.key |
sudo gpg --yes --dearmor --output /usr/share/keyrings/intel-graphics.gpg &&
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/intel-graphics.gpg] https://repositories.intel.com/gpu/ubuntu jammy/lts/2350 unified" |
sudo tee /etc/apt/sources.list.d/intel-gpu-jammy.list
2. Model does not seem to be using GPU
This line here seems to tell me that it's using the CPU instead of the GPU. Looking at the CPU usage with top and the duration for transcribing anything confirms that theory. I have not found out how to fix that.

/usr/local/lib/python3.10/dist-packages/whisper/transcribe.py:117: UserWarning: FP16 is not supported on CPU; using FP32 instead

Any updates? Did you make any progress? I am encountering the same errors.

[rbrowning85]

any updates or progress?

[Ian-Blockmans]

For anyone still interested in running on intel gpu's i made a fork and pull request #353. If anyone wants to test it here is the link.

[rbrowning85]

For anyone still interested in running on intel gpu's i made a fork and pull request #353. If anyone wants to test it here is the link.

Yes, I have still been searching for options to utilize my Intel Arc GPU. I will check out the link and let you know if I have any questions or issues. Thanks.

[rbrowning85]

For anyone still interested in running on intel gpu's i made a fork and pull request #353. If anyone wants to test it here is the link.

I had to update Dockerfile.intel to Dockerfile.gpu.xpu

Then I received the following error. I am not sure how to resolve this one:

 => ERROR [stage-2 11/11] RUN poetry install --extras xpu
------
 > [stage-2 11/11] RUN poetry install --extras xpu:
1.886 
1.886 Poetry could not find a pyproject.toml file in /app or its parents
------
Dockerfile.gpu.xpu:38

--------------------

  36 |     

  37 |     RUN poetry config virtualenvs.in-project true

  38 | >>> RUN poetry install --extras xpu

  39 |     

  40 |     EXPOSE 9000

--------------------

failed to solve: process "/bin/bash -c poetry install --extras xpu" did not complete successfully: exit code: 1

[Ian-Blockmans]

Ok first one is on me I forgot to change the name in the yaml file that is fixed. For the second one if the pyproject.toml file is in the same directory as the dockerfile it should run. Did you clone the entire repo? If you just cant get it to work just use the package i uploaded to github.
example compose i run:

services:
  openai-whisper-asr-webservice-intel:
    image: ghcr.io/ian-blockmans/whisper-asr-webservice-intel:latest
    environment:
      - ASR_ENGINE=openai_whisper
      - ASR_DEVICE=xpu
    ports:
      - 9000:9000
    devices:
      - /dev/dri:/dev/dri

[rbrowning85]

Ok first one is on me I forgot to change the name in the yaml file that is fixed. For the second one if the pyproject.toml file is in the same directory as the dockerfile it should run. Did you clone the entire repo? If you just cant get it to work just use the package i uploaded to github. example compose i run:

services:
  openai-whisper-asr-webservice-intel:
    image: ghcr.io/ian-blockmans/whisper-asr-webservice-intel:latest
    environment:
      - ASR_ENGINE=openai_whisper
      - ASR_DEVICE=xpu
    ports:
      - 9000:9000
    devices:
      - /dev/dri:/dev/dri

that worked, but now i am getting the following error in the logs:

Traceback (most recent call last):
  File "/app/.venv/bin/whisper-asr-webservice", line 3, in <module>
    from app.webservice import start
ModuleNotFoundError: No module named 'app.webservice'

[Ian-Blockmans]

that worked, but now i am getting the following error in the logs:

Traceback (most recent call last):
  File "/app/.venv/bin/whisper-asr-webservice", line 3, in <module>
    from app.webservice import start
ModuleNotFoundError: No module named 'app.webservice'

Ok my bad I should have been clearer. You have to clone the repo to build the image. I am guessing you did not. Am I right?