Why DALI model accumulate GPU memory when sending requests?

[gjegoj]

Hi, I encountered a problem - when I send requests to the model, it starts reserving memory on the GPU, in general there is nothing wrong with this, but after I stop sending requests - the memory is not cleared, moreover, if I start sending requests again, the memory continues to be reserved and not cleared. I use locust to send requests to the model, but the same situation happens if I use perf_analyzer, just used this example (https://github.com/triton-inference-server/dali_backend/tree/main/docs/examples/perf_analyzer)

dali.py:

import nvidia.dali as dali
import nvidia.dali.types as types
from nvidia.dali.plugin.triton import autoserialize

IMAGE_SIZE = (512, 512)
MEAN = dali.fn.constant(fdata=[0.485, 0.456, 0.406], shape=(1, 1, 3)) * 255
STD = dali.fn.constant(fdata=[0.229, 0.224, 0.225], shape=(1, 1, 3)) * 255


@autoserialize
@dali.pipeline_def(batch_size=8, num_threads=4, device_id=0)
def pipe():
    images = dali.fn.external_source(device="cpu", name="DALI_INPUT")
    images = dali.fn.decoders.image(images, device="mixed", output_type=types.RGB)
    images = dali.fn.resize(images, resize_x=IMAGE_SIZE[0], resize_y=IMAGE_SIZE[1], interp_type=types.INTERP_LINEAR)
    images = dali.fn.normalize(images, scale=1, mean=MEAN, stddev=STD, shift=0)
    images = dali.fn.transpose(images, perm=[2, 0, 1])
    images = dali.fn.cast(images, dtype=types.FLOAT)

    return images

config.pbtxt:

name: "dali"
backend: "dali"
max_batch_size: 8
dynamic_batching {
  preferred_batch_size: [ 2, 4, 8 ]
  max_queue_delay_microseconds: 1000
}
input [
    {
        name: "DALI_INPUT"
        data_type: TYPE_UINT8
        dims: [ -1 ]
        allow_ragged_batch: true
    }
]
output [
    {
        name: "DALI_OUTPUT"
        data_type: TYPE_FP32
        dims: [ 3, 512, 512 ]
    }
]
instance_group [
  {
    count: 1
    kind: KIND_GPU
    gpus: [ 0 ]
  }
]
optimization {
  graph : {
    level : 1
  }
}

model repo:

models/
├── dali
│   ├── 1
│   │   └── dali.py
│   └── config.pbtxt

I use Triton RELEASE=25.02
The command to run Triton:

docker build -t triton:dev -f ./docker/Dockerfile .
docker run --rm -it --init --shm-size=1GB -p 8000:8000 -p 8001:8001 -p 8002:8002 triton:dev

...
tritonserver \
		--model-repository=/models \
		--allow-metrics=1 \
		--log-verbose=0 \
		--metrics-config summary_latencies=true \
		--pinned-memory-pool-byte-size=1073741824 \
  		--cuda-memory-pool-byte-size=0:1073741824

before sending requests:

after sending 20k requests:

sending another 20k requests:

I noticed that when I change the parameter device="mixed" to device="cpu" at dali.fn.decoders.image(images, device="mixed", output_type=types.RGB), the problem disappears. But at the same time the RPS of the model falls.

[szalpal]

Hi @gjegoj !

Could you provide more information about the requests you are sending? From what you've pointed out in the last paragraph, the effect you're observing comes from ImageDecoder. What are the shapes of the images you're sending to the input?

When operating, DALI C++ layer reserves some GPU memory (e.g. for CUDA context) and does not release it, unless the libdali.so is unloaded. I believe you might be facing this effect here. However, please provide more specs regarding the input sizes so I could assess if the memory consumption you're facing is expected

[gjegoj]

Hi @gjegoj !

Could you provide more information about the requests you are sending? From what you've pointed out in the last paragraph, the effect you're observing comes from ImageDecoder. What are the shapes of the images you're sending to the input?

When operating, DALI C++ layer reserves some GPU memory (e.g. for CUDA context) and does not release it, unless the libdali.so is unloaded. I believe you might be facing this effect here. However, please provide more specs regarding the input sizes so I could assess if the memory consumption you're facing is expected

Thank you @szalpal for your reply.
Here is an dummy example of how and what requests I send. Sometimes I can send in batches, sometimes one picture at a time, the resolutions in the batch can be different from 720x1280 to 3840x2160.

import requests
import numpy as np
from tritonclient.http.aio import InferenceServerClient
from tritonclient.http.aio import InferInput, InferRequestedOutput

urls = [
    'https://raw.githubusercontent.com/triton-inference-server/dali_backend/refs/heads/main/client/images/monkey-653705_1920.jpg',
    'https://raw.githubusercontent.com/triton-inference-server/dali_backend/refs/heads/main/client/images/baboon-3089012_1280.jpg'
]

image_bytes_list = [requests.get(url).content for url in urls]

def pad_image_bytes(image_bytes_list: list[bytes]) -> np.ndarray:
    max_size = max(len(img) for img in image_bytes_list)
    padded_images = [img.ljust(max_size, b"\x00") for img in image_bytes_list]
    return np.stack([np.frombuffer(img, dtype=np.uint8) for img in padded_images])

image_data = pad_image_bytes(image_bytes_list)  # image_data.shape -> (2, 556326)

inputs = [InferInput("DALI_INPUT", image_data.shape, 'UINT8').set_data_from_numpy(image_data)]
outputs = [InferRequestedOutput("DALI_OUTPUT")]

triton_server = InferenceServerClient("0.0.0.0:8000")
result = await triton_server.infer(
    model_name='dali',
    inputs=inputs,
    outputs=outputs,
)

[gjegoj]

Hi @szalpal
Did you check my previous message? Any comments regarding my problem?