UWSGI container high memory usage on Kubernetes with 2.53.1

[goat-ssh]

Bug description
We've upgraded v2.50.4 -> 2.53.1. Our usage pattern have not changed and the container has quite some spikes

Expected behavior
Memory usage is stable

Deployment method (select with an X)

• Docker Compose
• Kubernetes
• GoDojo

Environment information

• DefectDojo version 2.53.1
• k8s 1.33 AKS

Screenshots

Additional context (optional)

helm values (chart 1.9.1):

celery:
  beat:
    resources:
      limits:
        cpu: 500m
      requests:
        cpu: 100m
  worker:
    resources:
      limits:
        cpu: 500m
      requests:
        cpu: 100m
createPostgresqlSecret: true
createSecret: true
createValkeySecret: true
django:
  ingress:
    annotations:
      nginx.ingress.kubernetes.io/proxy-body-size: 50M
    ingressClassName: nginx
  nginx:
    resources:
      limits:
        cpu: 200m
      requests:
        cpu: 100m
  uwsgi:
    enableDebug: true
    resources:
      limits:
        cpu: 500m
        memory: 1Gi
      requests:
        cpu: 100m
global:
  postgresql:
    auth:
      password: ddddd
host: dojo.xyz

[valentijnscholten]

Quite a lot has changed between 2.50.x and 2.53.x. Do you have any idea which version introduced it?

The upgrade notes lists the biggest changes:

• https://docs.defectdojo.com/en/open_source/upgrading/2.51/
• https://docs.defectdojo.com/en/open_source/upgrading/2.52/
• https://docs.defectdojo.com/en/open_source/upgrading/2.53/
• and docs: add dedupe batching note to 2.53 upgrade notes #13914
• python 3.12 -> 3.13

Is there anything in there that rings a bell or is typically used a lot in your instance?

Biggest changes in the uwsgi container code in these releases have been about improving performance. But some of them could lead to a little more memory use. The graph however doesn't show "massive" memory use. More like a tradtional saw tooth pattern I know from Java applications. We don't get many memory issue reports and I have never looked into how to debug them.

Is the memory usage causing issues, or is it all recovering by itself and not hurting performance?

[goat-ssh]

We've 1Gi limit set, this should cause the saw tooth pattern. I'll increase it again

[b3ndill]

Just wanted to chime in on this issue. We are seeing a similar pattern on the memory usage of the UWSGI container with a fresh Defect Dojo setup:

• Defect Dojo version: 2.52.3
• Deployment type: GKE via Helm chart
• Helm chart version: 1.8.3
• Database type: External PostgreSQL via Cloud SQL

After the initial setup we noticed that the Django pod kept restarting which was due to the UWSGI container hitting the memory limit and Kubernetes killing it. We raised the memory limit to 2GB but it only prolonged the lifetime to around two days as can be seen on the graph below. After some research into the issue, we suspected that this was related to "unbound" file descriptors on the host systems. We tried setting the django.uwsgi.appSettings.maxFd parameter to 1024 but it didn't change the memory usage trajectory.

We haven't started using Defect Dojo apart from setting up external authentication and logging in.

We're currently running the UWSGI container without any resource limits to see if it balances out somewhere but it's approaching 2.5GB and isn't showing any signs of stopping... 😛

[valentijnscholten]

I don't see this behaviour in my local docker compose based instance. The DD Pro team also doesn't see it in the Pro SAAS instances. But those instances user tcpPorts based health checks which might confirm the HTTP health checks are causing the memory issues as mentioned by others on Slack: https://owasp.slack.com/archives/C2P5BA8MN/p1763049449585229

[b3ndill]

@valentijnscholten that might be the cause of the memory "leak".

I tested this by converting the uwsgi container liveness probe to a TCP probe as well as removing the nginx container readiness probe.
The memory utilization is now stable at around 500MiB (with no usage), before it would grow by 20-30MiB per hour under the same circumstances.