Http calls staying blocked for way too long

[victornoel]

Describe the bug

Hi, I am using kubernetes-client with vertx, and executing blocking calls to k8s in their dedicated worker threads.

From time to time, I'm seeing some of those calls staying blocked waiting for results for way too long, vertx tells me the thread stayed blocked for 104526455ms aka 30h.

So either I'm misunderstanding something or there is a bug somewhere because I would expect the http client default timeouts to be triggered way before :)

The stack trace when it happens:

Thread Thread[vert.x-worker-thread-9,5,main] has been blocked for 104526455 ms, time limit is 60000 ms","stack":"io.vertx.core.VertxException: Thread blocked
	at jdk.internal.misc.Unsafe.park(Unknown Source)
	at java.util.concurrent.locks.LockSupport.park(Unknown Source)
	at java.util.concurrent.CompletableFuture$Signaller.block(Unknown Source)
	at java.util.concurrent.ForkJoinPool.unmanagedBlock(Unknown Source)
	at java.util.concurrent.ForkJoinPool.managedBlock(Unknown Source)
	at java.util.concurrent.CompletableFuture.waitingGet(Unknown Source)
	at java.util.concurrent.CompletableFuture.get(Unknown Source)
	at i.f.k.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:491)
	at i.f.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:451)
	at i.f.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:419)
	at i.f.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:98)
        ...

There is of course many such errors in the logs since the start of that blocking call 30h before that one above.

One suspicious (but maybe completely unrelated) thing that happens kind of at the time corresponding to the start of that blocking call, is that I see one line in the process output (but not a well formatted log from logback or anything) with the single word "pause" in it. As far as I could find info about this, I would say it's some gc logs from the JVM saying it paused the JVM?

Fabric8 Kubernetes Client version

7.1.0

Steps to reproduce

No idea :)

Expected behavior

Blocking call failing after some timeout (or even better, finishing successfully if it should have of course)

Runtime

Kubernetes (vanilla)

Kubernetes API Server version

1.32

Environment

GKE

Fabric8 Kubernetes Client Logs

Additional context

No response

[shawkins]

Please try to reproduce this with 7.4+. In particular #7163 was addressed.

[victornoel]

@shawkins interesting! Will do and report back then :) thx

[victornoel]

@shawkins looks like this did fix the problem indeed, in the last 3 weeks I've seen only at most 5s of blocking with from time to time 2s of blocking but not often.

Thanks!