Kubernetes 1.33.x upgrade/fresh install not working

[vgeorgiev69]

/kind bug

What steps did you take and what happened:
[A clear and concise description of what the bug is.]

I've successfully upgraded my clusters from 1.31 to 1.32.9. The problem occurred when upgrading from 1.32.9 to 1.33.x
Did a complete fresh cluster with all of the 1.33.x versions - all of them fail. These are the logs I'm seeing:

Sep 23 13:12:25 test-control-plane-thfpm kubelet[2727]: E0923 13:12:25.351537    2727 event.go:368] "Unable to write event (may retry after sleeping)" err="Post \"https://91.98.81.42:6443/api/v1/namespaces/default/events\": dial tcp 91.98.81.42:6443: connect: connection refused" event="&Event{ObjectMeta:{test-control-plane-thfpm.1867eb757653ac72  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:test-control-plane-thfpm,UID:test-control-plane-thfpm,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:test-control-plane-thfpm,},FirstTimestamp:2025-09-23 13:11:09.240364146 +0000 UTC m=+0.080328188,LastTimestamp:2025-09-23 13:11:09.240364146 +0000 UTC m=+0.080328188,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:test-control-plane-thfpm,}"
Sep 23 13:12:25 test-control-plane-thfpm kubelet[2727]: E0923 13:12:25.369536    2727 kubelet.go:3305] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"test-control-plane-thfpm\" not found" node="test-control-plane-thfpm"
Sep 23 13:12:25 test-control-plane-thfpm kubelet[2727]: I0923 13:12:25.369693    2727 scope.go:117] "RemoveContainer" containerID="a8a8fc1ad37a7fc7b0d281d01c4bbda500edc94edef19d4614bd3a6625671aa9"
Sep 23 13:12:25 test-control-plane-thfpm kubelet[2727]: E0923 13:12:25.369947    2727 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-apiserver\" with CrashLoopBackOff: \"back-off 40s restarting failed container=kube-apiserver pod=kube-apiserver-test-control-plane-thfpm_kube-system(41c00f3f721fc8b46f670503139a929d)\"" pod="kube-system/kube-apiserver-test-control-plane-thfpm" podUID="41c00f3f721fc8b46f670503139a929d"
Sep 23 13:12:28 test-control-plane-thfpm kubelet[2727]: E0923 13:12:28.289761    2727 reflector.go:200] "Failed to watch" err="failed to list *v1.CSIDriver: Get \"https://91.98.81.42:6443/apis/storage.k8s.io/v1/csidrivers?limit=500&resourceVersion=0\": dial tcp 91.98.81.42:6443: connect: connection refused" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 23 13:12:28 test-control-plane-thfpm kubelet[2727]: E0923 13:12:28.369017    2727 kubelet.go:3305] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"test-control-plane-thfpm\" not found" node="test-control-plane-thfpm"
Sep 23 13:12:29 test-control-plane-thfpm kubelet[2727]: E0923 13:12:29.328745    2727 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"test-control-plane-thfpm\" not found"
Sep 23 13:12:31 test-control-plane-thfpm kubelet[2727]: E0923 13:12:31.869714    2727 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://91.98.81.42:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/test-control-plane-thfpm?timeout=10s\": dial tcp 91.98.81.42:6443: connect: connection refused" interval="7s"
Sep 23 13:12:32 test-control-plane-thfpm kubelet[2727]: I0923 13:12:32.057992    2727 kubelet_node_status.go:75] "Attempting to register node" node="test-control-plane-thfpm"
Sep 23 13:12:32 test-control-plane-thfpm kubelet[2727]: E0923 13:12:32.058555    2727 kubelet_node_status.go:107] "Unable to register node with API server" err="Post \"https://91.98.81.42:6443/api/v1/nodes\": dial tcp 91.98.81.42:6443: connect: connection refused" node="test-control-plane-thfpm"
Sep 23 13:12:35 test-control-plane-thfpm kubelet[2727]: E0923 13:12:35.352320    2727 event.go:368] "Unable to write event (may retry after sleeping)" err="Post \"https://91.98.81.42:6443/api/v1/namespaces/default/events\": dial tcp 91.98.81.42:6443: connect: connection refused" event="&Event{ObjectMeta:{test-control-plane-thfpm.1867eb757653ac72  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:test-control-plane-thfpm,UID:test-control-plane-thfpm,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:test-control-plane-thfpm,},FirstTimestamp:2025-09-23 13:11:09.240364146 +0000 UTC m=+0.080328188,LastTimestamp:2025-09-23 13:11:09.240364146 +0000 UTC m=+0.080328188,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:test-control-plane-thfpm,}"

What did you expect to happen:
To have 1.33.x working as the previous versions.

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

• cluster-api-provider-hetzner version: 1.0.6, the rest are 1.10.6
• Kubernetes version: (use kubectl version) Client Version: v1.32.1 / Kustomize Version: v5.5.0 / Server Version: v1.32.9
• OS (e.g. from /etc/os-release): Ubuntu 22 and Ubuntu 24.04

[guettli]

@vgeorgiev69 can you please show the output of:

go run github.com/guettli/check-conditions@latest all

KUBECONFIG need to be connected to the mgt-cluster.

[vgeorgiev69]

@vgeorgiev69 can you please show the output of:

go run github.com/guettli/check-conditions@latest all

KUBECONFIG need to be connected to the mgt-cluster.

Sure, this is the output:

  default hcloudmachinetemplates k8s-mgmt-control-plane Condition Ready=False HetznerSecretUnreachable "could not find HetznerSecret" (4560h29m8s)
  default hcloudmachinetemplates k8s-mgmt-md-0 Condition Ready=False HetznerSecretUnreachable "could not find HetznerSecret" (4560h29m8s)
  default machinehealthchecks k8s-mgmt-md-0-unhealthy-5m Condition ExternalRemediationTemplateAvailable=False ExternalRemediationTemplateNotFound "failed to retrieve HCloudRemediationTemplate default/worker-remediation-request: HCloudRemediationTemplate.infrastructure.cluster.x-k8s.io \"worker-remediation-request\" not found" (4560h21m33s)
Checked 583 conditions of 664 resources of 95 types. Duration: 3.531s
exit status 1

Let me know what else I can provide. Thanks!

[guettli]

HetznerSecretUnreachable "could not find HetznerSecret"

A secret is needed for the Hetzner Robot API. This secret seems to be missing.

Please read the docs about that.

[vgeorgiev69]

I'm not seeing how this is relevant, unless something majorly changed with that robot API in 1.33.

I've upgraded my clusters from 1.30.x to 1.32.9 without any issues. As mentioned in the description, even a new, fresh cluster with any 1.33 version has the same kube-api errors as listed above.

[guettli]

I guess you are right. The Condition is unhealty for "ages".

I tried that:

❯ curl --insecure https://91.98.81.42:6443
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
  "reason": "Forbidden",
  "details": {},
  "code": 403
}

does that work for you?

Is the controller able to connect that URL?

[guettli]

I can see that the api-server is still running:

❯ curl -k https://91.98.81.42:6443/version
{
  "major": "1",
  "minor": "32",
  "gitVersion": "v1.32.9",
  "gitCommit": "cea7087b31eb788b75934d769a28f058ab309318",
  "gitTreeState": "clean",
  "buildDate": "2025-09-09T19:37:59Z",
  "goVersion": "go1.23.12",
  "compiler": "gc",
  "platform": "linux/amd64"
}

Why is the caph controller not able to connect to that server/port?

[vgeorgiev69]

I guess you are right. The Condition is unhealty for "ages".

I tried that:

❯ curl --insecure https://91.98.81.42:6443
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {},
"status": "Failure",
"message": "forbidden: User "system:anonymous" cannot get path "/"",
"reason": "Forbidden",
"details": {},
"code": 403
}

does that work for you?

Is the controller able to connect that URL?

I'm having the same output from that curl.

To be honest I'm not really sure why does it try to connect to that IP. That IP is a control plane node from a completely different cluster. I'd appreciate any input on why this is happening.

I will do more debugging to understand why is it trying to connect to there even though this is a newly created cluster generated with the usual clusterctl command from the syself documentation.

Edit: One thing to add here - my mgmt cluster is on version 1.31.7, can this be the culprit? Although this shouldn't be an issue as I've successfully upgraded multiple clusters from 1.30 to 1.32 without issues.

[vgeorgiev69]

These are the logs from the newly created cluster. Now it's trying to access itself with the same issue.

I believe something is wrong in the logic here, because it's not trying to connect to the actual LB API which is created by default (and on all of my other clusters) which is currently with IP 91.98.15.104

2025-09-26T08:31:08.013795+00:00 test-control-plane-7jcwd kubelet[3381]: E0926 08:31:08.013336    3381 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://5.75.233.113:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/test-control-plane-7jcwd?timeout=10s\": dial tcp 5.75.233.113:6443: connect: connection refused" interval="7s"
2025-09-26T08:31:08.177347+00:00 test-control-plane-7jcwd kubelet[3381]: I0926 08:31:08.177225    3381 kubelet_node_status.go:75] "Attempting to register node" node="test-control-plane-7jcwd"
2025-09-26T08:31:08.177676+00:00 test-control-plane-7jcwd kubelet[3381]: E0926 08:31:08.177593    3381 kubelet_node_status.go:107] "Unable to register node with API server" err="Post \"https://5.75.233.113:6443/api/v1/nodes\": dial tcp 5.75.233.113:6443: connect: connection refused" node="test-control-plane-7jcwd"
2025-09-26T08:31:10.443193+00:00 test-control-plane-7jcwd kubelet[3381]: E0926 08:31:10.442676    3381 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"test-control-plane-7jcwd\" not found"
2025-09-26T08:31:11.941880+00:00 test-control-plane-7jcwd kubelet[3381]: E0926 08:31:11.941127    3381 event.go:368] "Unable to write event (may retry after sleeping)" err="Post \"https://5.75.233.113:6443/api/v1/namespaces/default/events\": dial tcp 5.75.233.113:6443: connect: connection refused" event="&Event{ObjectMeta:{test-control-plane-7jcwd.1868c7cff0340001  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:test-control-plane-7jcwd,UID:test-control-plane-7jcwd,APIVersion:,ResourceVersion:,FieldPath:,},Reason:InvalidDiskCapacity,Message:invalid capacity 0 on image filesystem,Source:EventSource{Component:kubelet,Host:test-control-plane-7jcwd,},FirstTimestamp:2025-09-26 08:29:10.390276097 +0000 UTC m=+0.103304496,LastTimestamp:2025-09-26 08:29:10.390276097 +0000 UTC m=+0.103304496,Count:1,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:test-control-plane-7jcwd,}"
2025-09-26T08:31:15.014784+00:00 test-control-plane-7jcwd kubelet[3381]: E0926 08:31:15.014157    3381 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://5.75.233.113:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/test-control-plane-7jcwd?timeout=10s\": dial tcp 5.75.233.113:6443: connect: connection refused" interval="7s"
2025-09-26T08:31:15.179543+00:00 test-control-plane-7jcwd kubelet[3381]: I0926 08:31:15.179440    3381 kubelet_node_status.go:75] "Attempting to register node" node="test-control-plane-7jcwd"
2025-09-26T08:31:15.179944+00:00 test-control-plane-7jcwd kubelet[3381]: E0926 08:31:15.179831    3381 kubelet_node_status.go:107] "Unable to register node with API server" err="Post \"https://5.75.233.113:6443/api/v1/nodes\": dial tcp 5.75.233.113:6443: connect: connection refused" node="test-control-plane-7jcwd"
2025-09-26T08:31:15.472698+00:00 test-control-plane-7jcwd kubelet[3381]: E0926 08:31:15.472564    3381 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: Get \"https://5.75.233.113:6443/api/v1/services?fieldSelector=spec.clusterIP%21%3DNone&limit=500&resourceVersion=0\": dial tcp 5.75.233.113:6443: connect: connection refused" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
2025-09-26T08:31:16.452692+00:00 test-control-plane-7jcwd kubelet[3381]: E0926 08:31:16.452104    3381 kubelet.go:3305] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"test-control-plane-7jcwd\" not found" node="test-control-plane-7jcwd"
2025-09-26T08:31:16.452902+00:00 test-control-plane-7jcwd kubelet[3381]: I0926 08:31:16.452261    3381 scope.go:117] "RemoveContainer" containerID="e6de7e3ba0e087b996f5f066e27e3907ca470906fd35d501424140a24cec4a64"
2025-09-26T08:31:16.452968+00:00 test-control-plane-7jcwd kubelet[3381]: E0926 08:31:16.452475    3381 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-apiserver\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=kube-apiserver pod=kube-apiserver-test-control-plane-7jcwd_kube-system(2187cf51bb77a8424e962a7f523901e8)\"" pod="kube-system/kube-apiserver-test-control-plane-7jcwd" podUID="2187cf51bb77a8424e962a7f523901e8"
2025-09-26T08:31:16.491689+00:00 test-control-plane-7jcwd kubelet[3381]: E0926 08:31:16.491555    3381 reflector.go:200] "Failed to watch" err="failed to list *v1.CSIDriver: Get \"https://5.75.233.113:6443/apis/storage.k8s.io/v1/csidrivers?limit=500&resourceVersion=0\": dial tcp 5.75.233.113:6443: connect: connection refused" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
2025-09-26T08:31:17.923456+00:00 test-control-plane-7jcwd kubelet[3381]: E0926 08:31:17.922839    3381 certificate_manager.go:596] "Failed while requesting a signed certificate from the control plane" err="cannot create certificate signing request: Post \"https://5.75.233.113:6443/apis/certificates.k8s.io/v1/certificatesigningrequests\": dial tcp 5.75.233.113:6443: connect: connection refused" logger="kubernetes.io/kube-apiserver-client-kubelet.UnhandledError"

This is the kubelet status

● kubelet.service - kubelet: The Kubernetes Node Agent
     Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; preset: enabled)
    Drop-In: /usr/lib/systemd/system/kubelet.service.d
             └─10-kubeadm.conf
     Active: active (running) since Fri 2025-09-26 08:29:10 UTC; 4min 0s ago
       Docs: https://kubernetes.io/docs/
   Main PID: 3381 (kubelet)
      Tasks: 11 (limit: 4540)
     Memory: 30.5M (peak: 31.4M)
        CPU: 4.909s
     CGroup: /system.slice/kubelet.service
             └─3381 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config>

Sep 26 08:33:00 test-control-plane-7jcwd kubelet[3381]: E0926 08:33:00.916607    3381 event.go:368] "Unable to write event (may retry after sleeping)" err="Post \"https://5.>
Sep 26 08:33:07 test-control-plane-7jcwd kubelet[3381]: E0926 08:33:07.038361    3381 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://5.75.>
Sep 26 08:33:07 test-control-plane-7jcwd kubelet[3381]: I0926 08:33:07.216479    3381 kubelet_node_status.go:75] "Attempting to register node" node="test-control-plane-7jcwd"
Sep 26 08:33:07 test-control-plane-7jcwd kubelet[3381]: E0926 08:33:07.217996    3381 kubelet_node_status.go:107] "Unable to register node with API server" err="Post \"https>
Sep 26 08:33:09 test-control-plane-7jcwd kubelet[3381]: E0926 08:33:09.305597    3381 reflector.go:200] "Failed to watch" err="failed to list *v1.RuntimeClass: Get \"https:/>
Sep 26 08:33:10 test-control-plane-7jcwd kubelet[3381]: E0926 08:33:10.452032    3381 kubelet.go:3305] "No need to create a mirror pod, since failed to get node info from th>
Sep 26 08:33:10 test-control-plane-7jcwd kubelet[3381]: I0926 08:33:10.452138    3381 scope.go:117] "RemoveContainer" containerID="2fb6c5db6cb42bcbfe95d1db2ddcb378a83211b94d>
Sep 26 08:33:10 test-control-plane-7jcwd kubelet[3381]: E0926 08:33:10.452580    3381 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to>
Sep 26 08:33:10 test-control-plane-7jcwd kubelet[3381]: E0926 08:33:10.453464    3381 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" fo>
Sep 26 08:33:10 test-control-plane-7jcwd kubelet[3381]: E0926 08:33:10.925194    3381 event.go:368] "Unable to write event (may retry after sleeping)" err="Post \"https://5.>
lines 4-23/23 (END)

[guettli]

@vgeorgiev69 you can use that tool to dump all resources:

guettli/dumpall: Dump all Kubernetes resources into a directory structure

Then you can use grep/rg to see if the IP is somewhere in a manifest.

[vgeorgiev69]

@vgeorgiev69 you can use that tool to dump all resources:

guettli/dumpall: Dump all Kubernetes resources into a directory structure

Then you can use grep/rg to see if the IP is somewhere in a manifest.

Thanks, i'll try it out to see if i can further narrow down the issue.
My only assumption is either a bug in the syself provider or some major changes in 1.33 regarding the kube api.

The latest log output I pasted is showing that the node is asking itself where the API server is not/shouldn't be running, because I'm using the plain default config that is generated with clusterctl:

spec:
  controlPlaneEndpoint:
    host: ""
    port: 443
  controlPlaneLoadBalancer:
    region: fsn1
  controlPlaneRegions:
  - fsn1

I don't have any changes on the hetzner side, and I've been spinning clusters like this for a while now.