VRAM not released after client disconnect in sherpa-onnx-online-websocket-server

[Alekksander66]

Hello !

When using sherpa-onnx-online-websocket-server with CUDA provider, GPU VRAM usage keeps increasing after handling multiple WebSocket connections.

Even after a client finishes streaming (Done message sent, final result returned, connection closed), VRAM allocated by the model is not released. Over time, this leads to out-of-memory (OOM) errors or forces the server process to crash/restart.

This issue makes it impossible to run the server under heavy load (hundreds of concurrent streams), since VRAM usage grows linearly with the number of completed connections.

./bin/sherpa-onnx-online-websocket-server \
  --port=8080 \
  --num-work-threads=16 \
  --num-io-threads=8 \
  --tokens=./models/tokens.txt \
  --encoder=./models/encoder.onnx \
  --decoder=./models/decoder.onnx \
  --joiner=./models/joiner.onnx \
  --provider=cuda \
  --max-batch-size=128 \
  --loop-interval-ms=10

Actual behavior

VRAM usage keeps increasing after each client disconnect.
Even though connections are removed from connections_ in OnlineWebsocketDecoder::ProcessConnections, the GPU memory is not freed.
Eventually the server hits OOM and restarts.

Environment

sherpa-onnx version: latest

Build type: The binaries from wget https://github.com/k2-fsa/sherpa-onnx/releases/download/v1.12.13/sherpa-onnx-v1.12.13-cuda-12.x-cudnn-9.x-linux-x64-gpu.tar.bz2

CUDA version: 12.8

GPU: NVIDIA H100 80GB

OS: nvidia/cuda:12.8.1-cudnn-runtime-ubuntu22.04

The issue seems related to OnlineRecognizer / OnlineStream not freeing GPU state after InputFinished + removal from connections_.
I tried adding manual cleanup (resetting stream, clearing connections), but VRAM still accumulates.

On CPU provider, the memory is released correctly.
On CUDA provider, VRAM grows continuously with each completed stream.

[csukuangfj]

@dosubot

[dosubot]

There’s no official documentation or known issues about GPU VRAM management or CUDA memory leaks in sherpa-onnx. In ONNX Runtime and similar frameworks, GPU memory should be released when objects like streams or recognizers are destroyed and all references are cleared. If VRAM keeps growing after client disconnects, it’s likely that some objects or buffers are lingering in memory, or the CUDA provider isn’t freeing resources as expected.

To investigate, check that OnlineRecognizer and OnlineStream objects are actually destroyed after a connection closes, and confirm there are no lingering references in your code or in the server’s connection management. Also, monitor the ONNX Runtime CUDA provider for any known memory release issues. If possible, try explicit resource cleanup or process isolation for heavy loads.

If you want to dig deeper, reviewing the destruction and cleanup logic for streams and recognizers in the codebase is a good next step. If you find that objects are destroyed but VRAM is still not released, it may be worth checking ONNX Runtime’s CUDA provider for bugs or limitations.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}   

[Alekksander66]

I also notice that even when running with --provider=cuda, the server shows very high CPU usage.
This makes me wonder if inference is really happening on the GPU, or if only part of the pipeline is offloaded.

Is this expected? Should GPU execution significantly reduce CPU load, or is high CPU usage normal in the current implementation?

[csukuangfj]

@dosu

[dosubot]

There isn't official documentation on expected CPU usage or how much of the pipeline is GPU-accelerated with --provider=cuda. Inference itself (the neural network forward pass) should run on the GPU, but other parts—like audio preprocessing, feature extraction, websocket I/O, and orchestration—still use the CPU. High CPU usage can result from these steps, thread management, or data transfer overhead, especially under heavy load or with many concurrent streams.

If you want to confirm that inference is happening on the GPU, you can monitor GPU utilization (e.g., with nvidia-smi) during active streams. To pinpoint CPU bottlenecks, profiling the server process (e.g., with perf or similar tools) can help identify which parts of the code are consuming the most CPU. High CPU usage is not unusual in mixed CPU/GPU pipelines, especially if non-inference tasks are significant or if batching is suboptimal.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}