Informer hasSynced() issue with Fabric8 Kubernetes client 7.4.0

[iampratapak]

Describe the bug

After upgrading the Kubernetes client to version 7.4.0, we have intermittently been seeing an informer sync issue.

Our liveness probe checks the health of our informers. If an informer's health is not good, the liveness probe fails, and the pod restarts. After upgrading the Fabric8 Kubernetes client to version 7.4.0, our service pod has been restarting several times. Upon investigation, we found that the health of some informers was not good.

We check if an informer is running(), watching() and hasSynced() to determine if its health is good.

However, after downgrading the Kubernetes client to version 7.3.1, we are no longer seeing the service pod restart and the informers' health is also good.

I have attached the reproducer here. We can deploy this to any Kubernetes cluster to reproduce the issue. Just deploy it and let it run for three to four hours. You will then observe pod restarts several times due to a health failure caused by the informer issue.

cc: @manusa

Fabric8 Kubernetes Client version

7.4.0

Steps to reproduce

1. Upgrade the Kubernetes Client to version 7.4.0
2. Use a SharedIndexInformer as part of the application's health check
3. Configure the Kubernetes liveness probe to check the SharedIndexInformer's health
4. Deploy the application in a Kubernetes cluster and let it run for three to four hours

Expected behavior

SharedIndexInformer health should not fail in Kubernetes client version 7.4.0

Runtime

other (please specify in additional context)

Kubernetes API Server version

1.32

Environment

Linux, other (please specify in additional context)

Fabric8 Kubernetes Client Logs

HealthResponseImpl[status=UP, details={PodInformer hasSynced=true, PodInformer isRunning=true, PodInformer isWatching=true}]
HealthResponseImpl[status=DOWN, details={PodInformer hasSynced=true, PodInformer isRunning=true, PodInformer isWatching=false}]
Sep 23, 2025 11:08:00 AM io.helidon.Main shutdown
INFO: Shutdown requested by JVM shutting down
Sep 23, 2025 11:08:00 AM io.helidon.webserver.ServerListener listen
INFO: [0x53e218d9] @default socket closed.
Sep 23, 2025 11:08:00 AM io.helidon.Main shutdown
INFO: Shutdown finished

Additional context

livenessProbe:
  failureThreshold: 1
  httpGet:
    path: /health
    port: 8080
    scheme: HTTP
  initialDelaySeconds: 30
  periodSeconds: 15
  successThreshold: 1
  timeoutSeconds: 2

Version:

kubectl version:

Client Version: v1.32.1
Kustomize Version: v5.5.0
Server Version: v1.32.1

We are using Oracle Kubernetes Engine (OKE) clusters to deploy our applications

kubernetes-client-informer-sync-issue.zip

[shawkins]

The watching check was refined and is now more acurate. You should be careful using that as part of a health check as the status is ephemeral and is supposed to be self-healing. You will need quite a few failures in row to be indicative of a problem, and there's no guarentee that restarting will fix it.

[iampratapak]

@shawkins
Should we stop including the Informer health check as part of our liveness probe, or out of the options below, which ones do you recommend we keep as part of the health check and which ones should we exclude?

isRunning()

isWatching()

hasSynced()

Alternatively, should we increase the failureThreshold to a higher value, like 20?

[shawkins]

If you never stop your informers, then isRunning=false would be a hard, and unlikely, error. hasSynced once true, won't flip to false again, so it belongs more to a startup rather than a health probe.

watches have a built-in timeout, you may also see connectivity issues to your api-server, or the api-server itself may be restarted. In all those cases isWatching will become temporarily false, you can either increase the failure threshold or remove it from the health check.

[iampratapak]

Thanks, @shawkins. I will modify our health check as per your suggestion and run it for a couple of days. I'll get back to you if I notice any further issues.