Resource reconciliation fails on kubernetes v1.33+

[piotr-cwiklinski-ds]

I've deployed kaap using helm chart version 0.3.4 on local kubernetes cluster with api version 1.33.2 kindest/node:v1.33.2
Kaap operator pod is running, but no zookeeper and booker pods are created.
Checking the logs I found stack trace which seem to be related to kubernetes java client issue kubernetes-client/java#4054

11:09:14 ERROR [com.dat.oss.kaa.con.AbstractController] (ReconcilerExecutor-pulsar-zk-controller-96) Error during reconciliation for resource zookeepers.kaap.oss.datastax.com with name pulsar-zookeeper: An error has occurred.: io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred.
	at io.fabric8.kubernetes.client.KubernetesClientException.copyAsCause(KubernetesClientException.java:238)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:536)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:570)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:554)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.restCall(OperationSupport.java:798)
	at io.fabric8.kubernetes.client.impl.BaseClient.getVersionInfo(BaseClient.java:265)
	at io.fabric8.kubernetes.client.impl.KubernetesClientImpl.getVersion(KubernetesClientImpl.java:599)
	at io.fabric8.openshift.client.impl.OpenShiftClientImpl.getKubernetesVersion(OpenShiftClientImpl.java:562)
	at io.fabric8.kubernetes.client.NamespacedKubernetesClientAdapter.getKubernetesVersion(NamespacedKubernetesClientAdapter.java:192)
	at com.datastax.oss.kaap.controllers.BaseResourcesFactory.getVersion(BaseResourcesFactory.java:265)
	at com.datastax.oss.kaap.controllers.BaseResourcesFactory.isPdbSupported(BaseResourcesFactory.java:255)
	at com.datastax.oss.kaap.controllers.BaseResourcesFactory.createPodDisruptionBudgetIfEnabled(BaseResourcesFactory.java:624)
	at com.datastax.oss.kaap.controllers.zookeeper.ZooKeeperResourcesFactory.patchPodDisruptionBudget(ZooKeeperResourcesFactory.java:224)
	at com.datastax.oss.kaap.controllers.zookeeper.ZooKeeperController.patchAll(ZooKeeperController.java:63)
	at com.datastax.oss.kaap.controllers.zookeeper.ZooKeeperController.patchResources(ZooKeeperController.java:52)
	at com.datastax.oss.kaap.controllers.zookeeper.ZooKeeperController.patchResources(ZooKeeperController.java:32)
	at com.datastax.oss.kaap.controllers.AbstractController.reconcile(AbstractController.java:139)
	at com.datastax.oss.kaap.controllers.AbstractController.reconcile(AbstractController.java:62)
	at com.datastax.oss.kaap.controllers.zookeeper.ZooKeeperController_ClientProxy.reconcile(Unknown Source)
	at io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:145)
	at io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:103)
	at io.javaoperatorsdk.operator.monitoring.micrometer.MicrometerMetrics.lambda$timeControllerExecution$0(MicrometerMetrics.java:86)
	at io.micrometer.core.instrument.composite.CompositeTimer.record(CompositeTimer.java:69)
	at io.javaoperatorsdk.operator.monitoring.micrometer.MicrometerMetrics.timeControllerExecution(MicrometerMetrics.java:84)
	at io.javaoperatorsdk.operator.processing.Controller.reconcile(Controller.java:102)
	at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.reconcileExecution(ReconciliationDispatcher.java:139)
	at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleReconcile(ReconciliationDispatcher.java:119)
	at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleDispatch(ReconciliationDispatcher.java:89)
	at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleExecution(ReconciliationDispatcher.java:62)
	at io.javaoperatorsdk.operator.processing.event.EventProcessor$ReconcilerExecutor.run(EventProcessor.java:406)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred.
	at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:129)
	at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:122)
	at io.fabric8.kubernetes.client.utils.Serialization.unmarshal(Serialization.java:257)
	at io.fabric8.kubernetes.client.utils.Serialization.unmarshal(Serialization.java:366)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.lambda$handleResponse$0(OperationSupport.java:603)
	at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.lambda$retryWithExponentialBackoff$2(OperationSupport.java:643)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
	at io.fabric8.kubernetes.client.http.ByteArrayBodyHandler.onBodyDone(ByteArrayBodyHandler.java:52)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
	at io.fabric8.kubernetes.client.okhttp.OkHttpClientImpl$OkHttpAsyncBody.doConsume(OkHttpClientImpl.java:133)
	... 3 more
Caused by: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "emulationMajor" (class io.fabric8.kubernetes.client.VersionInfo), not marked as ignorable (9 known properties: "goVersion", "gitTreeState", "platform", "minor", "gitVersion", "gitCommit", "buildDate", "compiler", "major"])
 at [Source: (BufferedInputStream); line: 4, column: 22] (through reference chain: io.fabric8.kubernetes.client.VersionInfo["emulationMajor"])
	at com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException.from(UnrecognizedPropertyException.java:61)
	at com.fasterxml.jackson.databind.DeserializationContext.handleUnknownProperty(DeserializationContext.java:1132)
	at com.fasterxml.jackson.databind.deser.std.StdDeserializer.handleUnknownProperty(StdDeserializer.java:2202)
	at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownProperty(BeanDeserializerBase.java:1705)
	at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownVanilla(BeanDeserializerBase.java:1683)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:320)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:177)
	at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:323)
	at com.fasterxml.jackson.databind.ObjectReader._bindAndClose(ObjectReader.java:2105)
	at com.fasterxml.jackson.databind.ObjectReader.readValue(ObjectReader.java:1481)
	at io.fabric8.kubernetes.client.utils.Serialization.unmarshal(Serialization.java:253)
	... 19 more

[ahabel-wob]

We also see something similar on v1.33.4-gke.1134000 and kaap:0.3.3 versions.

10:37:15 ERROR [com.dat.oss.kaa.con.AbstractController] (ReconcilerExecutor-pulsar-bk-controller-86) Error during reconciliation for resource bookkeepers.kaap.oss.datastax.com with name pricing-pulsar-stg-bookkeeper...

Caused by: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "emulationMajor" (class io.fabric8.kubernetes.client.VersionInfo), not marked as ignorable (9 known properties: "goVersion", "gitTreeState", "platform", "minor", "gitVersion", "gitCommit", "buildDate", "compiler", "major"])  

Also related: fabric8io/kubernetes-client#7037

Update:
To reproduce replace the following contents in AbstractK3sContainer and run the tests. UnrecognizedPropertyException will be thrown.

// Change:
public static final KubernetesImageSpec<K3sContainerVersion> K3S_IMAGE =
            new KubernetesImageSpec<>(K3sContainerVersion.VERSION_1_31_0)
                    .withImage("rancher/k3s:v1.31.6-k3s1");

// To: (don't mind the old K3sContainerVersion, there is no 1_33 to use in the enum.
public static final KubernetesImageSpec<K3sContainerVersion> K3S_IMAGE =
            new KubernetesImageSpec<>(K3sContainerVersion.VERSION_1_31_0)
                    .withImage("rancher/k3s:v1.33.5-k3s1");

To fix it we need to upgrade to the latest kubernetes-client and quarkus packages. Unfortunately this is no trivial task, as they also contain breaking changes in the API.