Can use gpu when the audio data capture and transcribe?

[ffli]

Hello,
when i use the runtimespeechreconginzer in my pc, the language model is medium, my cpu which size is 16g will be 100% used. I found the whispper support use cuda gpu to accelerate, if i can use the gpu in this plugin?

[gtreshchev]

Hello. The whisper.cpp library supports the following options for acceleration:

• NVIDIA GPU support via cuBLAS
• OpenCL GPU support via CLBlast
• BLAS CPU support via OpenBLAS

However, integrating these libraries on my part would require dynamic linking and building them specifically for the supported target platforms, which would definitely complicate cross-compatibility. If you are working on a platform that supports one of these libraries, you can include them manually and accordingly modify the plugin to enable GPU acceleration.

[ffli]

Hello. The whisper.cpp library supports the following options for acceleration:

• NVIDIA GPU support via cuBLAS
• OpenCL GPU support via CLBlast
• BLAS CPU support via OpenBLAS

However, integrating these libraries on my part would require dynamic linking and building them specifically for the supported target platforms, which would definitely complicate cross-compatibility. If you are working on a platform that supports one of these libraries, you can include them manually and accordingly modify the plugin to enable GPU acceleration.

Hello, My environment is NVIDIA GPU, and i have update the gpu driver to support cuBLAS. If i use the plugin you latest update, can enable the gpu acceleration. Other settings or config should i change or add?

[gtreshchev]

There should be several changes required to enable that support in the plugin, such as the needed ggml files and changes to the Build.cs. I'll be thinking about how to achieve this smoothly for the supported platforms, but anyway, if it's not possible to achieve universal support, I will create some placeholders that developers can override and use if they have those libraries downloaded.

[ffli]

There should be several changes required to enable that support in the plugin, such as the needed ggml files and changes to the Build.cs. I'll be thinking about how to achieve this smoothly for the supported platforms, but anyway, if it's not possible to achieve universal support, I will create some placeholders that developers can override and use if they have those libraries downloaded.

Thanks. If should change the build.cs file, it means the plugin will be rebuilt. And the ggml files are the ggml module library or the ggml dynamic library?

[gtreshchev]

It would require installing the mentioned libraries in the system itself or including separate dynamic libraries for the target platform, and also including the needed ggml files to properly handle GPU acceleration: https://github.com/ggerganov/ggml/tree/master/src

I believe I can add some placeholders at the current stage to allow plugin users to include the libraries and enable GPU acceleration support manually easily, will consider this at some point.

[ffli]

It would require installing the mentioned libraries in the system itself or including separate dynamic libraries for the target platform, and also including the needed ggml files to properly handle GPU acceleration: https://github.com/ggerganov/ggml/tree/master/src

I believe I can add some placeholders at the current stage to allow plugin users to include the libraries and enable GPU acceleration support manually easily, will consider this at some point.

Ok, thank you.

[ScioEstasPotenco]

It will be great if you can add an option in a plugin's menu to support Cuda GPU as an option. For now, your plugin is best for speech recognition compared to other plugins.

[gtreshchev]

It will be great if you can add an option in a plugin's menu to support Cuda GPU as an option. For now, your plugin is best for speech recognition compared to other plugins.

Thanks! I can manually build the library for CUDA support and include an option in the plugin settings, but this would significantly complicate cross-compatibility since the library would need to be built for each supported platform.

And I'm not certain if it would be understandable and feasible enough for users to manually build the CUDA library for their specific platform. Regardless, I will try to find the best way to proceed, although not completely certain at the moment. Feel free to share your thoughts.

[ScioEstasPotenco]

You can create a separate GitHub folder for the CUDA version of this plugin or It will be helpful to write step-by-step instructions on how to build this specific plugin for GPU use.

[gtreshchev]

The recognition speed should now be much faster than before, so GPU acceleration is less prioritized for now

[ScioEstasPotenco]

I have tested with the medium and large models. Still, it takes time. I hope that you can make an option to include an NVidia GPU. I use this plugin for creating an AI avatar. With the CPU version of this plugin, it takes around 20 s to recognize one simple sentence in another language. Please, if you have time it will be great to make the GPU option available.

Thank you.

[gigiwotou]

I have tested on 3070 and 1660 using the whisper.dll inside whisper-cublas-12.2.0-bin-x64.zip, and found that the GPU is not faster than the CPU in recognition, instead, I feel that the CPU is faster.
I feel that hallucination has a greater impact on recognition efficiency.

[gtreshchev]

On Windows, the Fab version of the plugin now includes Vulkan GPU acceleration, introduced recently, which significantly improves recognition time: https://www.fab.com/listings/00ffc308-d7f9-4142-ac4c-4aeaa75ab54b