[BUG]: Intermittent `malloc(): unaligned tcache chunk detected` when launching a fastAPI server with `ddtrace-run`.

[marechaux]

Tracer Version(s)

3.12.2

Python Version(s)

3.12.11

Pip Version(s)

25.0.1

Bug Report

When upgrading our dependencies, we started to observe server hangs under load, always preceded by:

malloc(): unaligned tcache chunk detected

After some investigation, I determined that launching the service with ddtrace-run correlates with the hang; removing the runner eliminates it.

Initial symptom (before extra diagnostics)

• App: FastAPI server calling external APIs.
• Behavior in production: after ~1 hour of traffic, the process logs malloc(): unaligned tcache chunk detected and then hangs; subsequent requests fail (connection refused).
• Not reproducible locally/dev environment.

Differential behavior (runner vs in-code tracing)

• Failing: start with ddtrace-run → reproducible in production under load (typically within ~1 hour).

• Stable: start without ddtrace-run and enable tracing in code:

  import ddtrace
  ddtrace.patch_all()

  → stable for several days under the same production load.

Reproduction Code

I don’t have a minimal reproduction; this has only been observable under real prod load. The differential behavior (ddtrace-run vs. in code ddtrace.patch_all()) is consistent.

Error Logs

To make the failure occur earlier and to get actionable traces, I ran with:

PYTHONFAULTHANDLER=1
GLIBC_TUNABLES=glibc.malloc.tcache_max=0
MALLOC_PERTURB_=255
MALLOC_CHECK_=3

With these settings, the same scenario no longer hangs silently: it aborts/segfaults and produces:

• precise glibc allocator errors (double free or corruption, (!prev), unsorted chunks), and
• full Python faulthandler dumps for all threads on SIGABRT/SEGV.

2025-08-26 14:54:23.380000+00:00 malloc(): unaligned tcache chunk detected
2025-08-26 15:14:53.946000+00:00 Exception terminating connection <AdaptedConnection <asyncpg.connection.Connection object at 0x7f90dde4f2f0>>
2025-08-26 15:14:53.952000+00:00 Task exception was never retrievedfuture: <Task finished name='Task-10389' coro=<AsyncSession.close() done, defined at /app/.venv/lib/python3.12/site-packages/sqlalchemy/ext/asyncio/session.py:1016> exception=InterfaceError("(sqlalchemy.dialects.postgresql.asyncpg.InterfaceError) <class 'asyncpg.exceptions._base.InterfaceError'>: cannot call Transaction.rollback(): the underlying connection is closed")>
2025-08-26 15:24:35.656000+00:00 malloc(): unaligned tcache chunk detected
2025-08-26 15:24:36.657000+00:00 malloc(): unaligned tcache chunk detected
2025-08-26 17:32:34.658000+00:00 Unclosed client sessionclient_session: <aiohttp.client.ClientSession object at 0x7f6fd3115160>
2025-08-26 17:32:34.659000+00:00 Unclosed connectorconnections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7f6feeaa9670>, 19926.706405191)])']connector: <aiohttp.connector.TCPConnector object at 0x7f6fd3115c70>
2025-08-26 22:00:17.087000+00:00 Traceback (most recent call last): File "/app/.venv/lib/python3.12/site-packages/starlette/routing.py", line 701, in lifespan await receive() File "/app/.venv/lib/python3.12/site-packages/uvicorn/lifespan/on.py", line 137, in receive return await self.receive_queue.get() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/asyncio/queues.py", line 158, in get await getterasyncio.exceptions.CancelledError
2025-08-27 07:03:24.953000+00:00 <no Python frame>
2025-08-27 07:03:24.953000+00:00 Thread 0x00007fb01e8e56c0 (most recent call first):
2025-08-27 07:03:24.953000+00:00 <no Python frame>
2025-08-27 07:03:24.953000+00:00 Thread 0x00007fafd0b506c0 (most recent call first):
2025-08-27 07:03:24.953000+00:00 <no Python frame>
2025-08-27 07:03:24.953000+00:00 <no Python frame>
[output truncated, see joined full output]
2025-08-27 09:43:04.050000+00:00 Thread 0x00007f16a292f6c0 (most recent call first):
2025-08-27 09:43:04.050000+00:00 Thread 0x00007f164eafd6c0 (most recent call first):
2025-08-27 09:43:04.050000+00:00 Thread 0x00007f16a1f2e6c0 (most recent call first):
2025-08-27 09:43:04.050000+00:00 File "/app/.venv/lib/python3.12/site-packages/ddtrace/contrib/internal/futures/threading.py", line 41 in _wrap_execution
2025-08-27 09:43:04.050000+00:00 File "/app/.venv/lib/python3.12/site-packages/openai/_base_client.py", line 1915 in get_platform
2025-08-27 09:43:04.050000+00:00 File "/usr/local/lib/python3.12/platform.py", line 1245 in platform
2025-08-27 09:43:04.050000+00:00 File "/usr/local/lib/python3.12/platform.py", line 865 in __iter__
2025-08-27 09:43:04.050000+00:00 File "/usr/local/lib/python3.12/functools.py", line 998 in __get__
2025-08-27 09:43:04.050000+00:00 File "/usr/local/lib/python3.12/platform.py", line 860 in processor
2025-08-27 09:43:04.050000+00:00 File "/usr/local/lib/python3.12/platform.py", line 800 in get
2025-08-27 09:43:04.050000+00:00 File "/usr/local/lib/python3.12/platform.py", line 828 in from_subprocess
2025-08-27 09:43:04.050000+00:00 File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 59 in run
2025-08-27 09:43:04.050000+00:00 File "/usr/local/lib/python3.12/subprocess.py", line 466 in check_output
2025-08-27 09:43:04.050000+00:00 File "/app/.venv/lib/python3.12/site-packages/ddtrace/contrib/internal/trace_utils.py", line 297 in wrapper
2025-08-27 09:43:04.050000+00:00 File "/app/.venv/lib/python3.12/site-packages/ddtrace/contrib/internal/subprocess/patch.py", line 568 in _traced_subprocess_init
2025-08-27 09:43:04.050000+00:00 File "/app/.venv/lib/python3.12/site-packages/ddtrace/appsec/_common_module_patches.py", line 250 in popen_FD233052260D8B4D
2025-08-27 09:43:04.050000+00:00 File "/app/.venv/lib/python3.12/site-packages/ddtrace/appsec/_asm_request_context.py", line 364 in call_waf_callback
2025-08-27 09:43:04.050000+00:00 File "/app/.venv/lib/python3.12/site-packages/ddtrace/appsec/_processor.py", line 220 in waf_callable
2025-08-27 09:43:04.050000+00:00 File "/app/.venv/lib/python3.12/site-packages/ddtrace/appsec/_ddwaf/waf.py", line 187 in run
2025-08-27 09:43:04.050000+00:00 Fatal Python error: Aborted
2025-08-27 09:43:04.050000+00:00 <no Python frame>
2025-08-27 09:43:04.050000+00:00 File "/usr/local/lib/python3.12/subprocess.py", line 548 in run
2025-08-27 09:43:04.050000+00:00 File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 93 in _worker
2025-08-27 09:43:04.050000+00:00 File "/app/.venv/lib/python3.12/site-packages/ddtrace/appsec/_processor.py", line 300 in _waf_action
2025-08-27 09:43:04.050000+00:00 File "/usr/local/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
2025-08-27 09:43:04.050000+00:00 File "/usr/local/lib/python3.12/threading.py", line 1012 in run
2025-08-27 09:43:04.050000+00:00 Thread 0x00007f16a142d6c0 (most recent call first):
2025-08-27 09:43:04.050000+00:00 <no Python frame>
2025-08-27 09:43:04.050000+00:00 Thread 0x00007f168bfff6c0 (most recent call first):
2025-08-27 09:43:04.050000+00:00 <no Python frame>
2025-08-27 09:43:04.050000+00:00 Thread 0x00007f1654b676c0 (most recent call first):
2025-08-27 09:43:04.050000+00:00 File "/usr/local/lib/python3.12/threading.py", line 1032 in _bootstrap
2025-08-27 09:43:04.050000+00:00 File "/usr/local/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
2025-08-27 09:43:04.050000+00:00 File "/usr/local/lib/python3.12/threading.py", line 1012 in run
2025-08-27 09:43:04.050000+00:00 <no Python frame>
2025-08-27 09:43:04.050000+00:00 Thread 0x00007f164feff6c0 (most recent call first):
2025-08-27 09:43:04.050000+00:00 File "/usr/local/lib/python3.12/threading.py", line 1032 in _bootstrap
2025-08-27 09:43:04.050000+00:00 File "/usr/local/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
2025-08-27 09:43:04.050000+00:00 File "/app/.venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 956 in run
2025-08-27 09:43:04.050000+00:00 File "/usr/local/lib/python3.12/queue.py", line 171 in get
2025-08-27 09:43:04.050000+00:00 File "/usr/local/lib/python3.12/threading.py", line 355 in wait
2025-08-27 09:43:04.050000+00:00 Thread 0x00007f164f4fe6c0 (most recent call first):
2025-08-27 09:43:04.050000+00:00 File "/app/.venv/lib/python3.12/site-packages/datadog/dogstatsd/base.py", line 625 in _flush_thread_loop
2025-08-27 09:43:04.050000+00:00 File "/usr/local/lib/python3.12/threading.py", line 1032 in _bootstrap
2025-08-27 09:43:04.051000+00:00 File "/app/.venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 161 in _send
2025-08-27 09:43:04.051000+00:00 File "/app/.venv/lib/python3.12/site-packages/ddtrace/contrib/internal/asgi/middleware.py", line 442 in wrapped_send
2025-08-27 09:43:04.051000+00:00 File "/app/.venv/lib/python3.12/site-packages/ddtrace/contrib/internal/asgi/middleware.py", line 605 in _handle_http_response
2025-08-27 09:43:04.051000+00:00 File "/app/.venv/lib/python3.12/site-packages/ddtrace/internal/core/event_hub.py", line 108 in dispatch
2025-08-27 09:43:04.051000+00:00 File "/app/.venv/lib/python3.12/site-packages/ddtrace/appsec/_asm_request_context.py", line 596 in _call_waf
2025-08-27 09:43:04.051000+00:00 File "/app/.venv/lib/python3.12/site-packages/ddtrace/appsec/_asm_request_context.py", line 364 in call_waf_callback
2025-08-27 09:43:04.051000+00:00 Current thread 0x00007f16a641bb80 (most recent call first):
2025-08-27 09:43:04.051000+00:00 File "/app/.venv/lib/python3.12/site-packages/ddtrace/appsec/_processor.py", line 300 in _waf_action
2025-08-27 09:43:04.051000+00:00 File "/app/.venv/lib/python3.12/site-packages/ddtrace/appsec/_ddwaf/waf.py", line 187 in run
2025-08-27 09:43:04.051000+00:00 <no Python frame>
2025-08-27 09:43:04.051000+00:00 Thread 0x00007f16a48b86c0 (most recent call first):
2025-08-27 09:43:04.051000+00:00 <no Python frame>
2025-08-27 09:43:04.051000+00:00 Thread 0x00007f16a36496c0 (most recent call first):
2025-08-27 09:43:04.051000+00:00 <no Python frame>
2025-08-27 09:43:04.051000+00:00 File "/app/.venv/lib/python3.12/site-packages/ddtrace/appsec/_processor.py", line 220 in waf_callable
2025-08-27 09:43:04.051000+00:00 File "/app/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 214 in __call__
2025-08-27 09:43:04.051000+00:00 File "/usr/local/lib/python3.12/asyncio/runners.py", line 195 in run
2025-08-27 09:43:04.051000+00:00 File "/app/.venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 164 in __call__
2025-08-27 09:43:04.051000+00:00 Thread 0x00007f16a292f6c0 (most recent call first):
2025-08-27 09:43:04.051000+00:00 File "/app/.venv/bin/uvicorn", line 10 in <module>
2025-08-27 09:43:04.051000+00:00 File "/app/.venv/lib/python3.12/site-packages/click/core.py", line 1442 in __call__
2025-08-27 09:43:04.051000+00:00 File "/app/.venv/lib/python3.12/site-packages/click/core.py", line 1363 in main
2025-08-27 09:43:04.051000+00:00 File "/app/.venv/lib/python3.12/site-packages/click/core.py", line 1226 in invoke
2025-08-27 09:43:04.051000+00:00 File "/app/.venv/lib/python3.12/site-packages/click/core.py", line 794 in invoke
2025-08-27 09:43:04.051000+00:00 File "/app/.venv/lib/python3.12/site-packages/uvicorn/main.py", line 413 in main
2025-08-27 09:43:04.051000+00:00 File "/app/.venv/lib/python3.12/site-packages/uvicorn/main.py", line 580 in run
2025-08-27 09:43:04.051000+00:00 File "/app/.venv/lib/python3.12/site-packages/uvicorn/server.py", line 67 in run
2025-08-27 09:43:04.051000+00:00 File "/usr/local/lib/python3.12/asyncio/runners.py", line 118 in run
2025-08-27 09:43:04.051000+00:00 File "/app/.venv/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 409 in run_asgi
2025-08-27 09:43:04.051000+00:00 File "/app/.venv/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 60 in __call__
2025-08-27 09:43:04.051000+00:00 File "/app/.venv/lib/python3.12/site-packages/fastapi/applications.py", line 1054 in __call__
2025-08-27 09:43:04.051000+00:00 File "/app/.venv/lib/python3.12/site-packages/starlette/applications.py", line 113 in __call__
2025-08-27 09:43:04.051000+00:00 File "/app/.venv/lib/python3.12/site-packages/ddtrace/contrib/internal/asgi/middleware.py", line 491 in __call__
2025-08-27 09:43:04.051000+00:00 File "/app/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 185 in __call__
2025-08-27 09:43:04.051000+00:00 <no Python frame>
2025-08-27 09:43:04.051000+00:00 File "/usr/local/lib/python3.12/threading.py", line 1012 in run
2025-08-27 09:43:04.051000+00:00 <no Python frame>
2025-08-27 09:43:04.051000+00:00 File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 59 in run
2025-08-27 09:43:04.051000+00:00 File "/app/.venv/lib/python3.12/site-packages/ddtrace/contrib/internal/futures/threading.py", line 41 in _wrap_execution
2025-08-27 09:43:04.051000+00:00 File "/app/.venv/lib/python3.12/site-packages/openai/_base_client.py", line 1915 in get_platform
2025-08-27 09:43:04.051000+00:00 File "/usr/local/lib/python3.12/platform.py", line 1245 in platform
2025-08-27 09:43:04.051000+00:00 File "/usr/local/lib/python3.12/platform.py", line 865 in __iter__
2025-08-27 09:43:04.051000+00:00 File "/usr/local/lib/python3.12/functools.py", line 998 in __get__
2025-08-27 09:43:04.051000+00:00 File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 93 in _worker
2025-08-27 09:43:04.051000+00:00 File "/usr/local/lib/python3.12/platform.py", line 860 in processor
2025-08-27 09:43:04.051000+00:00 File "/usr/local/lib/python3.12/platform.py", line 828 in from_subprocess
2025-08-27 09:43:04.051000+00:00 File "/usr/local/lib/python3.12/subprocess.py", line 466 in check_output
2025-08-27 09:43:04.051000+00:00 File "/usr/local/lib/python3.12/subprocess.py", line 548 in run
2025-08-27 09:43:04.051000+00:00 File "/app/.venv/lib/python3.12/site-packages/ddtrace/contrib/internal/trace_utils.py", line 297 in wrapper
2025-08-27 09:43:04.051000+00:00 File "/app/.venv/lib/python3.12/site-packages/ddtrace/contrib/internal/subprocess/patch.py", line 568 in _traced_subprocess_init
2025-08-27 09:43:04.051000+00:00 File "/app/.venv/lib/python3.12/site-packages/ddtrace/appsec/_common_module_patches.py", line 250 in popen_FD233052260D8B4D
2025-08-27 09:43:04.051000+00:00 File "/usr/local/lib/python3.12/platform.py", line 800 in get
2025-08-27 09:43:04.051000+00:00 File "/usr/local/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
2025-08-27 09:43:04.051000+00:00 File "/usr/local/lib/python3.12/threading.py", line 1032 in _bootstrap
2025-08-27 09:43:04.051000+00:00 Thread 0x00007f164f4fe6c0 (most recent call first):
2025-08-27 09:43:04.051000+00:00 Thread 0x00007f16a142d6c0 (most recent call first):
2025-08-27 09:43:04.051000+00:00 <no Python frame>
2025-08-27 09:43:04.051000+00:00 Thread 0x00007f168bfff6c0 (most recent call first):
2025-08-27 09:43:04.051000+00:00 <no Python frame>
2025-08-27 09:43:04.051000+00:00 Thread 0x00007f1654b676c0 (most recent call first):
2025-08-27 09:43:04.051000+00:00 File "/usr/local/lib/python3.12/threading.py", line 1032 in _bootstrap
2025-08-27 09:43:04.051000+00:00 File "/usr/local/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
2025-08-27 09:43:04.051000+00:00 File "/usr/local/lib/python3.12/threading.py", line 1012 in run
2025-08-27 09:43:04.051000+00:00 File "/app/.venv/lib/python3.12/site-packages/datadog/dogstatsd/base.py", line 625 in _flush_thread_loop
2025-08-27 09:43:04.051000+00:00 Thread 0x00007f164feff6c0 (most recent call first):
2025-08-27 09:43:04.051000+00:00 File "/usr/local/lib/python3.12/threading.py", line 1032 in _bootstrap
2025-08-27 09:43:04.051000+00:00 File "/usr/local/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
2025-08-27 09:43:04.051000+00:00 File "/app/.venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 956 in run
2025-08-27 09:43:04.051000+00:00 File "/usr/local/lib/python3.12/queue.py", line 171 in get
2025-08-27 09:43:04.051000+00:00 File "/usr/local/lib/python3.12/threading.py", line 355 in wait
2025-08-27 09:43:04.051000+00:00 Thread 0x00007f16a1f2e6c0 (most recent call first):
2025-08-27 09:43:04.051000+00:00 Extension modules: ddtrace.internal._encoding, ddtrace.internal._threads, ddtrace.internal.telemetry.metrics_namespaces, ddtrace.internal._rand, ddtrace.internal._tagset, ddtrace.vendor.psutil._psutil_linux, ddtrace.vendor.psutil._psutil_posix, ddtrace.internal.datadog.profiling.ddup._ddup, ddtrace.internal.datadog.profiling.stack_v2._stack_v2, uvloop.loop, httptools.parser.parser, httptools.parser.url_parser, websockets.speedups, greenlet._greenlet, ujson, zstandard.backend_c, regex._regex, numpy._core._multiarray_umath, numpy.linalg._umath_linalg, charset_normalizer.md, simplejson._speedups, requests.packages.charset_normalizer.md, requests.packages.chardet.md, _cffi_backend, PIL._imaging, yaml._yaml, tornado.speedups, multidict._multidict, yarl._quoting_c, propcache._helpers_c, aiohttp._http_writer, aiohttp._http_parser, aiohttp._websocket.mask, aiohttp._websocket.reader_c, frozenlist._frozenlist, lxml._elementpath, lxml.etree, pyarrow.lib, sqlalchemy.cyextension.collections, sqlalchemy.cyextension.immutabledict, sqlalchemy.cyextension.processors, sqlalchemy.cyextension.resultproxy, sqlalchemy.cyextension.util, asyncpg.pgproto.pgproto, asyncpg.protocol.protocol, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._pcg64, numpy.random._mt19937, numpy.random._generator, numpy.random._philox, numpy.random._sfc64, numpy.random.mtrand, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.timestamps, pandas._libs.properties, pandas._libs.tslibs.offsets, pandas._libs.tslibs.strptime, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.lib, pyarrow._compute, pandas._libs.ops, pandas._libs.hashing, pandas._libs.arrays, pandas._libs.tslib, pandas._libs.sparse, pandas._libs.internals, pandas._libs.indexing, pandas._libs.index, pandas._libs.writers, pandas._libs.join, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.groupby, pandas._libs.json, pandas._libs.parsers, pandas._libs.testing (total: 95)

logs-ddtrace-failure.txt

Libraries in Use

aiobotocore==2.24.0
aiohappyeyeballs==2.6.1
aiohttp==3.12.15
aioitertools==0.12.0
aiosignal==1.4.0
altair==5.5.0
annotated-types==0.7.0
anyio==4.10.0
asn1crypto==1.5.1
asyncpg==0.30.0
attrs==25.3.0
authlib==1.5.2
beautifulsoup4==4.13.5
blinker==1.9.0
boto3==1.39.11
boto3-stubs==1.40.17
botocore==1.39.11
botocore-stubs==1.38.46
bytecode==0.16.2
cachetools==5.5.2
certifi==2025.8.3
cffi==1.17.1
cgroupspy==0.2.3
charset-normalizer==3.4.3
click==8.2.1
cramjam==2.11.0
cryptography==38.0.4
cssselect==1.3.0
datadog==0.52.1
dateparser==1.2.1
ddtrace==3.12.2
distro==1.9.0
dnspython==2.7.0
doctrine-aws-utils==0.2.19
doctrine-legal-graph-api==0.4.7
doctrine-positional-link-inserter==0.0.27
doctrine-python-utils==1.12.83
doctrine-workflow==2.2.5
duckdb==1.3.2
elastic-transport==8.17.1
elasticsearch==8.18.0
email-validator==2.2.0
envier==0.6.1
et-xmlfile==2.0.0
eval-type-backport==0.2.2
fastapi==0.116.1
fastapi-cli==0.0.8
fastapi-cloud-cli==0.1.5
fastparquet==2024.11.0
filelock==3.19.1
frozenlist==1.7.0
fsspec==2025.7.0
fuzzysearch==0.7.3
gitdb==4.0.12
gitpython==3.1.45
google-auth==2.40.3
google-genai==1.31.0
greenlet==3.2.4
h11==0.16.0
html2text==2025.4.15
httpcore==1.0.9
httptools==0.6.4
httpx==0.28.1
idna==3.10
importlib-metadata==8.7.0
invoke==2.2.0
itsdangerous==2.2.0
jinja2==3.1.6
jiter==0.10.0
jmespath==1.0.1
jsonpatch==1.33
jsonpointer==3.0.0
jsonref==1.1.0
jsonschema==4.25.1
jsonschema-specifications==2025.4.1
langchain-core==0.3.74
langchain-text-splitters==0.3.8
langsmith==0.4.17
levenshtein==0.27.1
lxml==6.0.1
lxml-html-clean==0.4.2
markdown-it-py==4.0.0
markupsafe==3.0.2
mdurl==0.1.2
mistralai==1.9.8
multidict==6.6.4
mypy-boto3-s3==1.40.0
mypy-boto3-sqs==1.40.17
narwhals==2.2.0
numpy==2.3.2
openai==1.101.0
openapi-pydantic==0.5.1
openpyxl==3.1.5
opentelemetry-api==1.36.0
orjson==3.11.2
packaging==25.0
pandas==2.3.2
pg8000==1.31.4
pillow==11.3.0
platformdirs==4.3.8
plotly==6.3.0
propcache==0.3.2
protobuf==6.32.0
psycopg==3.2.9
psycopg-binary==3.2.9
psycopg2-binary==2.9.10
pyarrow==21.0.0
pyasn1==0.6.1
pyasn1-modules==0.4.2
pycparser==2.22
pydantic==2.11.7
pydantic-core==2.33.2
pydantic-extra-types==2.10.5
pydantic-settings==2.10.1
pydeck==0.9.1
pygments==2.19.2
pyjwt==2.10.1
pyopenssl==22.0.0
python-dateutil==2.9.0.post0
python-dotenv==1.1.1
python-json-logger==3.3.0
python-multipart==0.0.20
pytz==2025.2
pyyaml==6.0.2
rapidfuzz==3.13.0
referencing==0.36.2
regex==2025.7.34
requests==2.32.5
requests-toolbelt==1.0.0
rich==14.1.0
rich-toolkit==0.15.0
rignore==0.6.4
rollbar==1.3.0
rpds-py==0.27.0
rsa==4.9.1
s3fs==2025.7.0
s3transfer==0.13.1
scramp==1.4.6
sentry-sdk==2.35.1
shellingham==1.5.4
simplejson==3.20.1
six==1.17.0
smmap==5.0.2
sniffio==1.3.1
snowflake-connector-python==3.17.2
sortedcontainers==2.4.0
soupsieve==2.7
sqids==0.5.2
sqlalchemy==2.0.43
starlette==0.47.3
streamlit==1.48.1
structlog==25.4.0
tenacity==9.1.2
termcolor==3.1.0
testing-common-database==2.0.3
testing-postgresql==1.3.0
text-unidecode==1.3
thefuzz==0.22.1
tiktoken==0.11.0
toml==0.10.2
tomlkit==0.13.3
tornado==6.5.2
tqdm==4.67.1
typer==0.16.1
types-awscrt==0.27.6
types-beautifulsoup4==4.12.0.20250516
types-html5lib==1.1.11.20250809
types-lxml==2025.8.25
types-protobuf==6.30.2.20250822
types-psycopg2==2.9.21.20250809
types-python-dateutil==2.9.0.20250822
types-pytz==2025.2.0.20250809
types-requests==2.31.0.6
types-s3transfer==0.13.0
types-urllib3==1.26.25.14
typing-extensions==4.15.0
typing-inspection==0.4.1
tzdata==2025.2
tzlocal==5.3.1
ujson==5.11.0
unidecode==1.4.0
universal-pathlib==0.2.6
urllib3==2.5.0
uvicorn==0.35.0
uvloop==0.21.0
watchdog==6.0.0
watchfiles==1.1.0
websockets==15.0.1
wrapt==1.17.3
yarl==1.20.1
zipp==3.23.0
zstandard==0.24.0

Operating System

Linux f1f0fd31f93b 6.10.14-linuxkit #1 SMP Fri Nov 29 17:22:03 UTC 2024 x86_64 GNU/Linux

[apenney]

I had the same issue @marechaux and I see you have sentry-sdk in your package list. When we had both enabled, we got the exact same WAF issues and malloc failures. Removing sentry immediately fixed this for us.

[marechaux]

Thanks for the pointer, @apenney — much appreciated.

We do have sentry-sdk as a transient dep (via fastapi[all]), but we don’t have code that initializes it (e.g., no sentry.init). I suspect it’s a different trigger on our side — though there may be a shared underlying issue manifesting differently.

[apenney]

Do you enable the DD_APPSEC_ENABLED stuff? That was the root cause for us - once we set that to false the issues went away. We've actually just disabled it everywhere because of yet another performance issue it's been causing us. Seems like it's pretty buggy, for the most part, so it might be worth checking if you can force disable it as a workaround in the meantime!

[marechaux]

Appreciate the heads-up, @apenney — my digging led me there too.

I tried to force-disable AppSec while still using ddtrace-run (DD_APPSEC_ENABLED=false, DD_REMOTE_CONFIGURATION_ENABLED=false, DD_REMOTE_CONFIG_ENABLED=0, DD_IAST_ENABLED=false), but it continued to crash the same way. With DD_TRACE_STARTUP_LOGS=true, I can also see the AppSec module still being loaded, which I don’t quite explain yet.

So far, the only workaround I found that fully stops the crash is skipping ddtrace-run and enabling tracing in code (ddtrace.patch_all()).

Did you have to do anything else to ensure AppSec stayed off under ddtrace-run, or is there another knob I should try?

[apenney]

I think that was enough for us - since I disabled appsec with just DD_APPSEC_ENABLED=false yesterday my problems have vanished (I have a second thread lockup issue that I think is the profiler instead, but... different problem).