K3S not starting after changes in /etc/network/interfaces #129
Description
Hello, as title says, I had to change the interface of my server since I changed its public IP, since now the k3s service is trying to use the old IP
I tried to start it manually using /usr/local/bin/k3s server --node-ip <new-server-ip> but issue remains the same
I have no idea what to do, found no K3S config file with a specific IP hard coded in that I could change.
What are the solutions ?
I replaced in the logs:
{MY-OLD-SERVER-IP} represents my old server ip (and therefor the new network interface name)
{MY-NEW-SERVER-IP} represents my new server ip (and therefor the new network interface name)
{SystemHostnameIG} represents my hostname
Oct 01 12:20:34 {SystemHostnameIG} systemd[1]: Starting k3s.service - Lightweight Kubernetes...
Oct 01 12:20:34 {SystemHostnameIG} sh[30600]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
Oct 01 12:20:34 {SystemHostnameIG} k3s[30604]: time="2025-10-01T12:20:34-04:00" level=info msg="Starting k3s v1.33.4+k3s1 (148243c4)"
Oct 01 12:20:34 {SystemHostnameIG} k3s[30604]: time="2025-10-01T12:20:34-04:00" level=info msg="Configuring sqlite3 database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s"
Oct 01 12:20:34 {SystemHostnameIG} k3s[30604]: time="2025-10-01T12:20:34-04:00" level=info msg="Configuring database table schema and indexes, this may take a moment..."
Oct 01 12:20:34 {SystemHostnameIG} k3s[30604]: time="2025-10-01T12:20:34-04:00" level=info msg="Database tables and indexes are up to date"
Oct 01 12:20:34 {SystemHostnameIG} k3s[30604]: time="2025-10-01T12:20:34-04:00" level=info msg="Kine available at unix://kine.sock"
Oct 01 12:20:34 {SystemHostnameIG} k3s[30604]: time="2025-10-01T12:20:34-04:00" level=info msg="Reconciling bootstrap data between datastore and disk"
Oct 01 12:20:35 {SystemHostnameIG} k3s[30604]: time="2025-10-01T12:20:35-04:00" level=info msg="Password verified locally for node {SystemHostnameIG}"
Oct 01 12:20:35 {SystemHostnameIG} k3s[30604]: time="2025-10-01T12:20:35-04:00" level=info msg="certificate CN={SystemHostnameIG} signed by CN=k3s-server-ca@1757698222: notBefore=2025-09-12 17:30:22 +0000 UTC notAfter=2026-10-01 16:20:35 +0000 UTC"
Oct 01 12:20:36 {SystemHostnameIG} k3s[30604]: time="2025-10-01T12:20:36-04:00" level=info msg="certificate CN=system:node:{SystemHostnameIG},O=system:nodes signed by CN=k3s-client-ca@1757698222: notBefore=2025-09-12 17:30:22 +0000 UTC notAfter=2026-10-01 16:20:36 +0000 UTC"
Oct 01 12:20:36 {SystemHostnameIG} k3s[30604]: time="2025-10-01T12:20:36-04:00" level=info msg="certificate CN=system:kube-proxy signed by CN=k3s-client-ca@1757698222: notBefore=2025-09-12 17:30:22 +0000 UTC notAfter=2026-10-01 16:20:36 +0000 UTC"
Oct 01 12:20:36 {SystemHostnameIG} k3s[30604]: time="2025-10-01T12:20:36-04:00" level=info msg="certificate CN=system:k3s-controller signed by CN=k3s-client-ca@1757698222: notBefore=2025-09-12 17:30:22 +0000 UTC notAfter=2026-10-01 16:20:36 +0000 UTC"
Oct 01 12:20:36 {SystemHostnameIG} k3s[30604]: time="2025-10-01T12:20:36-04:00" level=info msg="Module overlay was already loaded"
Oct 01 12:20:36 {SystemHostnameIG} k3s[30604]: time="2025-10-01T12:20:36-04:00" level=info msg="Module nf_conntrack was already loaded"
Oct 01 12:20:36 {SystemHostnameIG} k3s[30604]: time="2025-10-01T12:20:36-04:00" level=info msg="Module br_netfilter was already loaded"
Oct 01 12:20:36 {SystemHostnameIG} k3s[30604]: time="2025-10-01T12:20:36-04:00" level=info msg="Module iptable_nat was already loaded"
Oct 01 12:20:36 {SystemHostnameIG} k3s[30604]: time="2025-10-01T12:20:36-04:00" level=info msg="Module iptable_filter was already loaded"
Oct 01 12:20:36 {SystemHostnameIG} k3s[30604]: time="2025-10-01T12:20:36-04:00" level=warning msg="Failed to load kernel module nft-expr-counter with modprobe"
Oct 01 12:20:36 {SystemHostnameIG} k3s[30604]: time="2025-10-01T12:20:36-04:00" level=info msg="Logging containerd to /var/lib/rancher/k3s/agent/containerd/containerd.log"
Oct 01 12:20:36 {SystemHostnameIG} k3s[30604]: time="2025-10-01T12:20:36-04:00" level=info msg="Running containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml"
Oct 01 12:20:36 {SystemHostnameIG} k3s[30604]: time="2025-10-01T12:20:36-04:00" level=info msg="Polling for API server readiness: GET /readyz failed: the server is currently unable to handle the request"
Oct 01 12:20:37 {SystemHostnameIG} k3s[30604]: time="2025-10-01T12:20:37-04:00" level=info msg="containerd is now running"
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: time="2025-10-01T12:20:39-04:00" level=info msg="Connecting to proxy" url="wss://127.0.0.1:6443/v1-k3s/connect"
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: time="2025-10-01T12:20:39-04:00" level=info msg="Creating k3s-cert-monitor event broadcaster"
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: time="2025-10-01T12:20:39-04:00" level=info msg="Running kubelet --cloud-provider=external --config-dir=/var/lib/rancher/k3s/agent/etc/kubelet.conf.d --containerd=/run/k3s/containerd/containerd.sock --hostname-override={SystemHostnameIG} --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --node-ip={MY-NEW-SERVER-IP} --node-labels= --read-only-port=0"
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: time="2025-10-01T12:20:39-04:00" level=info msg="Handling backend connection request [{SystemHostnameIG}]"
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: time="2025-10-01T12:20:39-04:00" level=info msg="Remotedialer connected to proxy" url="wss://127.0.0.1:6443/v1-k3s/connect"
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: time="2025-10-01T12:20:39-04:00" level=info msg="Connection to etcd is ready"
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: time="2025-10-01T12:20:39-04:00" level=info msg="ETCD server is now running"
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: time="2025-10-01T12:20:39-04:00" level=info msg="Running kube-apiserver --advertise-address={MY-NEW-SERVER-IP} --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --etcd-servers=unix://kine.sock --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key"
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: time="2025-10-01T12:20:39-04:00" level=info msg="Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --leader-elect=false --profiling=false --secure-port=10259 --tls-cert-file=/var/lib/rancher/k3s/server/tls/kube-scheduler/kube-scheduler.crt --tls-private-key-file=/var/lib/rancher/k3s/server/tls/kube-scheduler/kube-scheduler.key"
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: I1001 12:20:39.564353 30604 options.go:249] external host was not specified, using {MY-NEW-SERVER-IP}
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: time="2025-10-01T12:20:39-04:00" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=,tokencleaner,-service,-route,-cloud-node-lifecycle --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --leader-elect=false --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/var/lib/rancher/k3s/server/tls/kube-controller-manager/kube-controller-manager.crt --tls-private-key-file=/var/lib/rancher/k3s/server/tls/kube-controller-manager/kube-controller-manager.key --use-service-account-credentials=true"
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: time="2025-10-01T12:20:39-04:00" level=info msg="Running cloud-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --bind-address=127.0.0.1 --cloud-config=/var/lib/rancher/k3s/server/etc/cloud-config.yaml --cloud-provider=k3s --cluster-cidr=10.42.0.0/16 --configure-cloud-routes=false --controllers=,-route --kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --leader-elect=false --leader-elect-resource-name=k3s-cloud-controller-manager --node-status-update-frequency=1m0s --profiling=false"
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: time="2025-10-01T12:20:39-04:00" level=info msg="Server node token is available at /var/lib/rancher/k3s/server/token"
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: I1001 12:20:39.566696 30604 server.go:155] Version: v1.33.4+k3s1
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: I1001 12:20:39.566772 30604 server.go:157] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: time="2025-10-01T12:20:39-04:00" level=info msg="To join server node to cluster: k3s server -s https://{MY-NEW-SERVER-IP}:6443 -t ${SERVER_NODE_TOKEN}"
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: time="2025-10-01T12:20:39-04:00" level=info msg="Agent node token is available at /var/lib/rancher/k3s/server/agent-token"
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: time="2025-10-01T12:20:39-04:00" level=info msg="To join agent node to cluster: k3s agent -s https://{MY-NEW-SERVER-IP}:6443 -t ${AGENT_NODE_TOKEN}"
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: time="2025-10-01T12:20:39-04:00" level=info msg="Wrote kubeconfig /etc/rancher/k3s/k3s.yaml"
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: time="2025-10-01T12:20:39-04:00" level=info msg="Run: k3s kubectl"
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: I1001 12:20:39.589225 30604 shared_informer.go:350] "Waiting for caches to sync" controller="node_authorizer"
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: I1001 12:20:39.595646 30604 shared_informer.go:350] "Waiting for caches to sync" controller="*generic.policySource[*k8s.io/api/admissionregistration/v1.ValidatingAdmissionPolicy,*k8s.io/api/admissionregistration/v1.ValidatingAdmissionPolicyBinding,k8s.io/apiserver/pkg/admission/plugin/policy/validating.Validator]"
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: I1001 12:20:39.603176 30604 plugins.go:157] Loaded 14 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,PodTopologyLabels,MutatingAdmissionPolicy,MutatingAdmissionWebhook.
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: I1001 12:20:39.603228 30604 plugins.go:160] Loaded 13 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,ClusterTrustBundleAttest,CertificateSubjectRestriction,ValidatingAdmissionPolicy,ValidatingAdmissionWebhook,ResourceQuota.
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: I1001 12:20:39.603662 30604 instance.go:233] Using reconciler: lease
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: I1001 12:20:39.610471 30604 handler.go:288] Adding GroupVersion apiextensions.k8s.io v1 to ResourceManager
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: W1001 12:20:39.610508 30604 genericapiserver.go:778] Skipping API apiextensions.k8s.io/v1beta1 because it has no resources.
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: I1001 12:20:39.648802 30604 cidrallocator.go:197] starting ServiceCIDR Allocator Controller
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: time="2025-10-01T12:20:39-04:00" level=error msg="Sending HTTP/1.1 503 response to 127.0.0.1:40884: runtime core not ready"
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: I1001 12:20:39.720487 30604 handler.go:288] Adding GroupVersion v1 to ResourceManager
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: I1001 12:20:39.720895 30604 apis.go:112] API group "internal.apiserver.k8s.io" is not enabled, skipping.
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: I1001 12:20:39.780069 30604 apis.go:112] API group "storagemigration.k8s.io" is not enabled, skipping.
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: I1001 12:20:39.854242 30604 apis.go:112] API group "resource.k8s.io" is not enabled, skipping.
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: I1001 12:20:39.867577 30604 handler.go:288] Adding GroupVersion authentication.k8s.io v1 to ResourceManager
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: W1001 12:20:39.867610 30604 genericapiserver.go:778] Skipping API authentication.k8s.io/v1beta1 because it has no resources.
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: W1001 12:20:39.867622 30604 genericapiserver.go:778] Skipping API authentication.k8s.io/v1alpha1 because it has no resources.
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: I1001 12:20:39.868198 30604 handler.go:288] Adding GroupVersion authorization.k8s.io v1 to ResourceManager
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: W1001 12:20:39.868222 30604 genericapiserver.go:778] Skipping API authorization.k8s.io/v1beta1 because it has no resources.
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: I1001 12:20:39.869337 30604 handler.go:288] Adding GroupVersion autoscaling v2 to ResourceManager
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: I1001 12:20:39.870078 30604 handler.go:288] Adding GroupVersion autoscaling v1 to ResourceManager
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: W1001 12:20:39.870107 30604 genericapiserver.go:778] Skipping API autoscaling/v2beta1 because it has no resources.
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: W1001 12:20:39.870113 30604 genericapiserver.go:778] Skipping API autoscaling/v2beta2 because it has no resources.
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: I1001 12:20:39.871492 30604 handler.go:288] Adding GroupVersion batch v1 to ResourceManager
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: W1001 12:20:39.871522 30604 genericapiserver.go:778] Skipping API batch/v1beta1 because it has no resources.
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: I1001 12:20:39.872229 30604 handler.go:288] Adding GroupVersion certificates.k8s.io v1 to ResourceManager
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: W1001 12:20:39.872252 30604 genericapiserver.go:778] Skipping API certificates.k8s.io/v1beta1 because it has no resources.
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: W1001 12:20:39.872259 30604 genericapiserver.go:778] Skipping API certificates.k8s.io/v1alpha1 because it has no resources.
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: I1001 12:20:39.872706 30604 handler.go:288] Adding GroupVersion coordination.k8s.io v1 to ResourceManager
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: W1001 12:20:39.872717 30604 genericapiserver.go:778] Skipping API coordination.k8s.io/v1beta1 because it has no resources.
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: W1001 12:20:39.872722 30604 genericapiserver.go:778] Skipping API coordination.k8s.io/v1alpha2 because it has no resources.
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: I1001 12:20:39.873224 30604 handler.go:288] Adding GroupVersion discovery.k8s.io v1 to ResourceManager
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: W1001 12:20:39.873243 30604 genericapiserver.go:778] Skipping API discovery.k8s.io/v1beta1 because it has no resources.
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: I1001 12:20:39.875789 30604 handler.go:288] Adding GroupVersion networking.k8s.io v1 to ResourceManager
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: W1001 12:20:39.875822 30604 genericapiserver.go:778] Skipping API networking.k8s.io/v1beta1 because it has no resources.
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: W1001 12:20:39.875830 30604 genericapiserver.go:778] Skipping API networking.k8s.io/v1alpha1 because it has no resources.
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: I1001 12:20:39.876316 30604 handler.go:288] Adding GroupVersion node.k8s.io v1 to ResourceManager
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: W1001 12:20:39.876331 30604 genericapiserver.go:778] Skipping API node.k8s.io/v1beta1 because it has no resources.
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: W1001 12:20:39.876337 30604 genericapiserver.go:778] Skipping API node.k8s.io/v1alpha1 because it has no resources.
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: I1001 12:20:39.877294 30604 handler.go:288] Adding GroupVersion policy v1 to ResourceManager
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: W1001 12:20:39.877315 30604 genericapiserver.go:778] Skipping API policy/v1beta1 because it has no resources.
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: I1001 12:20:39.879602 30604 handler.go:288] Adding GroupVersion rbac.authorization.k8s.io v1 to ResourceManager
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: W1001 12:20:39.879629 30604 genericapiserver.go:778] Skipping API rbac.authorization.k8s.io/v1beta1 because it has no resources.
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: W1001 12:20:39.879638 30604 genericapiserver.go:778] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: I1001 12:20:39.880213 30604 handler.go:288] Adding GroupVersion scheduling.k8s.io v1 to ResourceManager
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: W1001 12:20:39.880241 30604 genericapiserver.go:778] Skipping API scheduling.k8s.io/v1beta1 because it has no resources.
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: W1001 12:20:39.880249 30604 genericapiserver.go:778] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources.
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: I1001 12:20:39.882868 30604 handler.go:288] Adding GroupVersion storage.k8s.io v1 to ResourceManager
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: W1001 12:20:39.882917 30604 genericapiserver.go:778] Skipping API storage.k8s.io/v1beta1 because it has no resources.
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: W1001 12:20:39.882927 30604 genericapiserver.go:778] Skipping API storage.k8s.io/v1alpha1 because it has no resources.
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: time="2025-10-01T12:20:39-04:00" level=info msg="Running kube-proxy --cluster-cidr=10.42.0.0/16 --conntrack-max-per-core=0 --conntrack-tcp-timeout-close-wait=0s --conntrack-tcp-timeout-established=0s --healthz-bind-address=127.0.0.1 --hostname-override={SystemHostnameIG} --kubeconfig=/var/lib/rancher/k3s/agent/kubeproxy.kubeconfig --proxy-mode=iptables"
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: I1001 12:20:39.884362 30604 handler.go:288] Adding GroupVersion flowcontrol.apiserver.k8s.io v1 to ResourceManager
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: W1001 12:20:39.884403 30604 genericapiserver.go:778] Skipping API flowcontrol.apiserver.k8s.io/v1beta3 because it has no resources.
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: W1001 12:20:39.884412 30604 genericapiserver.go:778] Skipping API flowcontrol.apiserver.k8s.io/v1beta2 because it has no resources.
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: W1001 12:20:39.884417 30604 genericapiserver.go:778] Skipping API flowcontrol.apiserver.k8s.io/v1beta1 because it has no resources.
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: I1001 12:20:39.889152 30604 handler.go:288] Adding GroupVersion apps v1 to ResourceManager
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: W1001 12:20:39.889185 30604 genericapiserver.go:778] Skipping API apps/v1beta2 because it has no resources.
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: W1001 12:20:39.889197 30604 genericapiserver.go:778] Skipping API apps/v1beta1 because it has no resources.
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: I1001 12:20:39.891345 30604 handler.go:288] Adding GroupVersion admissionregistration.k8s.io v1 to ResourceManager
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: W1001 12:20:39.891382 30604 genericapiserver.go:778] Skipping API admissionregistration.k8s.io/v1beta1 because it has no resources.
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: W1001 12:20:39.891391 30604 genericapiserver.go:778] Skipping API admissionregistration.k8s.io/v1alpha1 because it has no resources.
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: I1001 12:20:39.891964 30604 handler.go:288] Adding GroupVersion events.k8s.io v1 to ResourceManager
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: W1001 12:20:39.891994 30604 genericapiserver.go:778] Skipping API events.k8s.io/v1beta1 because it has no resources.
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: I1001 12:20:39.897771 30604 handler.go:288] Adding GroupVersion apiregistration.k8s.io v1 to ResourceManager
Oct 01 12:20:39 {SystemHostnameIG} k3s[30604]: W1001 12:20:39.897811 30604 genericapiserver.go:778] Skipping API apiregistration.k8s.io/v1beta1 because it has no resources.
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.393509 30604 dynamic_cafile_content.go:161] "Starting controller" name="request-header::/var/lib/rancher/k3s/server/tls/request-header-ca.crt"
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.393510 30604 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/var/lib/rancher/k3s/server/tls/client-ca.crt"
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.393867 30604 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt::/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key"
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.393965 30604 secure_serving.go:211] Serving securely on 127.0.0.1:6444
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.393995 30604 tlsconfig.go:243] "Starting DynamicServingCertificateController"
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.394105 30604 remote_available_controller.go:411] Starting RemoteAvailability controller
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.394128 30604 cache.go:32] Waiting for caches to sync for RemoteAvailability controller
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.394146 30604 system_namespaces_controller.go:66] Starting system namespaces controller
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.394171 30604 controller.go:78] Starting OpenAPI AggregationController
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.394172 30604 dynamic_serving_content.go:135] "Starting controller" name="aggregator-proxy-cert::/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt::/var/lib/rancher/k3s/server/tls/client-auth-proxy.key"
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.394270 30604 controller.go:119] Starting legacy_token_tracking_controller
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.394280 30604 shared_informer.go:350] "Waiting for caches to sync" controller="configmaps"
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.394526 30604 customresource_discovery_controller.go:294] Starting DiscoveryController
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.394680 30604 controller.go:142] Starting OpenAPI controller
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.394721 30604 local_available_controller.go:156] Starting LocalAvailability controller
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.394741 30604 cache.go:32] Waiting for caches to sync for LocalAvailability controller
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.394764 30604 controller.go:80] Starting OpenAPI V3 AggregationController
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.394964 30604 default_servicecidr_controller.go:110] Starting kubernetes-service-cidr-controller
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.395001 30604 shared_informer.go:350] "Waiting for caches to sync" controller="kubernetes-service-cidr-controller"
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.395088 30604 apf_controller.go:377] Starting API Priority and Fairness config controller
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.395303 30604 controller.go:90] Starting OpenAPI V3 controller
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.395328 30604 cluster_authentication_trust_controller.go:459] Starting cluster_authentication_trust_controller controller
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.395337 30604 repairip.go:200] Starting ipallocator-repair-controller
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.395347 30604 shared_informer.go:350] "Waiting for caches to sync" controller="cluster_authentication_trust_controller"
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.395357 30604 shared_informer.go:350] "Waiting for caches to sync" controller="ipallocator-repair-controller"
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.395364 30604 nonstructuralschema_controller.go:195] Starting NonStructuralSchemaConditionController
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.395386 30604 apiapproval_controller.go:189] Starting KubernetesAPIApprovalPolicyConformantConditionController
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.395412 30604 crd_finalizer.go:269] Starting CRDFinalizer
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.394591 30604 aggregator.go:169] waiting for initial CRD sync...
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.394683 30604 gc_controller.go:78] Starting apiserver lease garbage collector
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.395660 30604 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/var/lib/rancher/k3s/server/tls/client-ca.crt"
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.395689 30604 apiservice_controller.go:100] Starting APIServiceRegistrationController
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.395703 30604 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.395742 30604 crdregistration_controller.go:114] Starting crd-autoregister controller
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.395751 30604 shared_informer.go:350] "Waiting for caches to sync" controller="crd-autoregister"
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.395342 30604 establishing_controller.go:81] Starting EstablishingController
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.395794 30604 dynamic_cafile_content.go:161] "Starting controller" name="request-header::/var/lib/rancher/k3s/server/tls/request-header-ca.crt"
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.395333 30604 naming_controller.go:299] Starting NamingConditionController
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.449944 30604 cidrallocator.go:301] created ClusterIP allocator for Service CIDR 10.43.0.0/16
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.489533 30604 shared_informer.go:357] "Caches are synced" controller="node_authorizer"
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.494857 30604 shared_informer.go:357] "Caches are synced" controller="configmaps"
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.494982 30604 cache.go:39] Caches are synced for RemoteAvailability controller
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.495053 30604 cache.go:39] Caches are synced for LocalAvailability controller
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.495242 30604 shared_informer.go:357] "Caches are synced" controller="kubernetes-service-cidr-controller"
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.495299 30604 default_servicecidr_controller.go:136] Shutting down kubernetes-service-cidr-controller
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.495301 30604 handler.go:288] Adding GroupVersion secretgenerator.mittwald.de v1alpha1 to ResourceManager
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.495338 30604 handler.go:288] Adding GroupVersion helm.cattle.io v1 to ResourceManager
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.495372 30604 handler.go:288] Adding GroupVersion traefik.io v1alpha1 to ResourceManager
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.495429 30604 handler.go:288] Adding GroupVersion hub.traefik.io v1alpha1 to ResourceManager
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.495500 30604 handler.go:288] Adding GroupVersion gateway.networking.k8s.io v1 to ResourceManager
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.495543 30604 shared_informer.go:357] "Caches are synced" controller="cluster_authentication_trust_controller"
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.495554 30604 shared_informer.go:357] "Caches are synced" controller="ipallocator-repair-controller"
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.495594 30604 apf_controller.go:382] Running API Priority and Fairness config worker
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.495607 30604 apf_controller.go:385] Running API Priority and Fairness periodic rebalancing process
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.495614 30604 handler.go:288] Adding GroupVersion gateway.networking.k8s.io v1beta1 to ResourceManager
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.495641 30604 handler.go:288] Adding GroupVersion k3s.cattle.io v1 to ResourceManager
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.495839 30604 cache.go:39] Caches are synced for APIServiceRegistrationController controller
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.495965 30604 shared_informer.go:357] "Caches are synced" controller="*generic.policySource[*k8s.io/api/admissionregistration/v1.ValidatingAdmissionPolicy,*k8s.io/api/admissionregistration/v1.ValidatingAdmissionPolicyBinding,k8s.io/apiserver/pkg/admission/plugin/policy/validating.Validator]"
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.495847 30604 shared_informer.go:357] "Caches are synced" controller="crd-autoregister"
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.496010 30604 policy_source.go:240] refreshing policies
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.496341 30604 aggregator.go:171] initial CRD sync complete...
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.496370 30604 autoregister_controller.go:144] Starting autoregister controller
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.496382 30604 cache.go:32] Waiting for caches to sync for autoregister controller
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.496393 30604 cache.go:39] Caches are synced for autoregister controller
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.516800 30604 handler_discovery.go:451] Starting ResourceDiscoveryManager
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.516950 30604 handler.go:288] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: W1001 12:20:40.516951 30604 handler_proxy.go:99] no RequestInfo found in the context
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: E1001 12:20:40.517045 30604 controller.go:146] "Unhandled Error" err=<
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: Error updating APIService "v1beta1.metrics.k8s.io" with err: failed to download v1beta1.metrics.k8s.io: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: , Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: > logger="UnhandledError"
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: I1001 12:20:40.521505 30604 controller.go:667] quota admission added evaluator for: leases.coordination.k8s.io
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: time="2025-10-01T12:20:40-04:00" level=info msg="Polling for API server readiness: GET /readyz failed: an error on the server ("[+]ping ok\n[+]log ok\n[+]etcd ok\n[+]etcd-readiness ok\n[+]informer-sync ok\n[+]poststarthook/start-apiserver-admission-initializer ok\n[+]poststarthook/generic-apiserver-start-informers ok\n[+]poststarthook/priority-and-fairness-config-consumer ok\n[+]poststarthook/priority-and-fairness-filter ok\n[+]poststarthook/storage-object-count-tracker-hook ok\n[+]poststarthook/start-apiextensions-informers ok\n[+]poststarthook/start-apiextensions-controllers ok\n[+]poststarthook/crd-informer-synced ok\n[+]poststarthook/start-system-namespaces-controller ok\n[+]poststarthook/start-cluster-authentication-info-controller ok\n[+]poststarthook/start-kube-apiserver-identity-lease-controller ok\n[+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok\n[+]poststarthook/start-legacy-token-tracking-controller ok\n[+]poststarthook/start-service-ip-repair-controllers ok\n[-]poststarthook/rbac/bootstrap-roles failed: reason withheld\n[-]poststarthook/scheduling/bootstrap-system-priority-classes failed: reason withheld\n[+]poststarthook/priority-and-fairness-config-producer ok\n[-]poststarthook/bootstrap-controller failed: reason withheld\n[+]poststarthook/start-kubernetes-service-cidr-controller ok\n[+]poststarthook/aggregator-reload-proxy-client-cert ok\n[+]poststarthook/start-kube-aggregator-informers ok\n[+]poststarthook/apiservice-status-local-available-controller ok\n[+]poststarthook/apiservice-status-remote-available-controller ok\n[+]poststarthook/apiservice-registration-controller ok\n[+]poststarthook/apiservice-discovery-controller ok\n[+]poststarthook/kube-apiserver-autoregistration ok\n[+]autoregister-completion ok\n[+]poststarthook/apiservice-openapi-controller ok\n[+]poststarthook/apiservice-openapiv3-controller ok\n[+]shutdown ok\nreadyz check failed") has prevented the request from succeeding"
Oct 01 12:20:40 {SystemHostnameIG} k3s[30604]: E1001 12:20:40.663780 30604 controller.go:102] Error removing old endpoints from kubernetes service: no API server IP addresses were listed in storage, refusing to erase all endpoints for the kubernetes Service
Oct 01 12:20:41 {SystemHostnameIG} k3s[30604]: I1001 12:20:41.421830 30604 storage_scheduling.go:111] all system priority classes are created successfully or already exist.
Oct 01 12:20:41 {SystemHostnameIG} k3s[30604]: W1001 12:20:41.502762 30604 handler_proxy.go:99] no RequestInfo found in the context
Oct 01 12:20:41 {SystemHostnameIG} k3s[30604]: W1001 12:20:41.502779 30604 handler_proxy.go:99] no RequestInfo found in the context
Oct 01 12:20:41 {SystemHostnameIG} k3s[30604]: E1001 12:20:41.502829 30604 controller.go:113] "Unhandled Error" err="loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: Error, could not get list of group versions for APIService" logger="UnhandledError"
Oct 01 12:20:41 {SystemHostnameIG} k3s[30604]: E1001 12:20:41.502889 30604 controller.go:102] "Unhandled Error" err=<
Oct 01 12:20:41 {SystemHostnameIG} k3s[30604]: loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to download v1beta1.metrics.k8s.io: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
Oct 01 12:20:41 {SystemHostnameIG} k3s[30604]: , Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
Oct 01 12:20:41 {SystemHostnameIG} k3s[30604]: > logger="UnhandledError"
Oct 01 12:20:41 {SystemHostnameIG} k3s[30604]: I1001 12:20:41.503953 30604 controller.go:109] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
Oct 01 12:20:41 {SystemHostnameIG} k3s[30604]: I1001 12:20:41.503959 30604 controller.go:126] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
Oct 01 12:20:42 {SystemHostnameIG} k3s[30604]: time="2025-10-01T12:20:42-04:00" level=info msg="Polling for API server readiness: GET /readyz failed: an error on the server ("[+]ping ok\n[+]log ok\n[+]etcd ok\n[+]etcd-readiness ok\n[+]informer-sync ok\n[+]poststarthook/start-apiserver-admission-initializer ok\n[+]poststarthook/generic-apiserver-start-informers ok\n[+]poststarthook/priority-and-fairness-config-consumer ok\n[+]poststarthook/priority-and-fairness-filter ok\n[+]poststarthook/storage-object-count-tracker-hook ok\n[+]poststarthook/start-apiextensions-informers ok\n[+]poststarthook/start-apiextensions-controllers ok\n[+]poststarthook/crd-informer-synced ok\n[+]poststarthook/start-system-namespaces-controller ok\n[+]poststarthook/start-cluster-authentication-info-controller ok\n[+]poststarthook/start-kube-apiserver-identity-lease-controller ok\n[+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok\n[+]poststarthook/start-legacy-token-tracking-controller ok\n[+]poststarthook/start-service-ip-repair-controllers ok\n[-]poststarthook/rbac/bootstrap-roles failed: reason withheld\n[+]poststarthook/scheduling/bootstrap-system-priority-classes ok\n[+]poststarthook/priority-and-fairness-config-producer ok\n[+]poststarthook/bootstrap-controller ok\n[+]poststarthook/start-kubernetes-service-cidr-controller ok\n[+]poststarthook/aggregator-reload-proxy-client-cert ok\n[+]poststarthook/start-kube-aggregator-informers ok\n[+]poststarthook/apiservice-status-local-available-controller ok\n[+]poststarthook/apiservice-status-remote-available-controller ok\n[+]poststarthook/apiservice-registration-controller ok\n[+]poststarthook/apiservice-discovery-controller ok\n[+]poststarthook/kube-apiserver-autoregistration ok\n[+]autoregister-completion ok\n[+]poststarthook/apiservice-openapi-controller ok\n[+]poststarthook/apiservice-openapiv3-controller ok\n[+]shutdown ok\nreadyz check failed") has prevented the request from succeeding"
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: Flag --containerd has been deprecated, This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: time="2025-10-01T12:20:44-04:00" level=info msg="Waiting for untainted node"
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: time="2025-10-01T12:20:44-04:00" level=info msg="Waiting for cloud-controller-manager privileges to become available"
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: time="2025-10-01T12:20:44-04:00" level=info msg="Kube API server is now running"
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: time="2025-10-01T12:20:44-04:00" level=info msg="k3s is up and running"
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: time="2025-10-01T12:20:44-04:00" level=info msg="Creating k3s-supervisor event broadcaster"
Oct 01 12:20:44 {SystemHostnameIG} systemd[1]: Started k3s.service - Lightweight Kubernetes.
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: I1001 12:20:44.586103 30604 event.go:389] "Event occurred" object="{SystemHostnameIG}" fieldPath="" kind="Node" apiVersion="" type="Normal" reason="CertificateExpirationOK" message="Node and Certificate Authority certificates managed by k3s are OK"
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: time="2025-10-01T12:20:44-04:00" level=info msg="Annotations and labels have already set on node: {SystemHostnameIG}"
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: I1001 12:20:44.588605 30604 server.go:526] "Kubelet version" kubeletVersion="v1.33.4+k3s1"
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: I1001 12:20:44.588772 30604 server.go:528] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: time="2025-10-01T12:20:44-04:00" level=info msg="Starting flannel with backend vxlan"
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: I1001 12:20:44.594624 30604 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/var/lib/rancher/k3s/agent/client-ca.crt"
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: I1001 12:20:44.603355 30604 server.go:1442] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: time="2025-10-01T12:20:44-04:00" level=info msg="Flannel found PodCIDR assigned for node {SystemHostnameIG}"
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: time="2025-10-01T12:20:44-04:00" level=info msg="The interface ens192 with ipv4 address {MY-NEW-SERVER-IP} will be used by flannel"
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: I1001 12:20:44.615554 30604 kube.go:139] Waiting 10m0s for node controller to sync
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: I1001 12:20:44.615577 30604 kube.go:469] Starting kube subnet manager
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: I1001 12:20:44.617724 30604 kube.go:490] Creating the node lease for IPv4. This is the n.Spec.PodCIDRs: [10.42.0.0/24]
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: I1001 12:20:44.629123 30604 server.go:778] "--cgroups-per-qos enabled, but --cgroup-root was not specified. defaulting to /"
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: I1001 12:20:44.629318 30604 server.go:839] "NoSwap is set due to memorySwapBehavior not specified" memorySwapBehavior="" FailSwapOn=false
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: I1001 12:20:44.629664 30604 container_manager_linux.go:267] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: I1001 12:20:44.629772 30604 container_manager_linux.go:272] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"{SystemHostnameIG}","RuntimeCgroupsName":"","SystemCgroupsName":"","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":false,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":null,"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":-1,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: I1001 12:20:44.630115 30604 topology_manager.go:138] "Creating topology manager with none policy"
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: I1001 12:20:44.630169 30604 container_manager_linux.go:303] "Creating device plugin manager"
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: I1001 12:20:44.630267 30604 state_mem.go:36] "Initialized new in-memory state store"
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: I1001 12:20:44.630593 30604 kubelet.go:480] "Attempting to sync node with API server"
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: I1001 12:20:44.630704 30604 kubelet.go:375] "Adding static pod path" path="/var/lib/rancher/k3s/agent/pod-manifests"
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: I1001 12:20:44.630809 30604 kubelet.go:386] "Adding apiserver pod source"
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: I1001 12:20:44.630899 30604 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: I1001 12:20:44.632089 30604 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="containerd" version="v2.0.5-k3s2" apiVersion="v1"
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: I1001 12:20:44.632599 30604 kubelet.go:935] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: I1001 12:20:44.633384 30604 watchdog_linux.go:99] "Systemd watchdog is not enabled"
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: I1001 12:20:44.633531 30604 server.go:1284] "Started kubelet"
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: I1001 12:20:44.633628 30604 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: I1001 12:20:44.633634 30604 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: I1001 12:20:44.634219 30604 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: I1001 12:20:44.635302 30604 server.go:317] "Adding debug handlers to kubelet server"
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: I1001 12:20:44.635453 30604 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: I1001 12:20:44.635465 30604 dynamic_serving_content.go:135] "Starting controller" name="kubelet-server-cert-files::/var/lib/rancher/k3s/agent/serving-kubelet.crt::/var/lib/rancher/k3s/agent/serving-kubelet.key"
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: I1001 12:20:44.635992 30604 volume_manager.go:297] "Starting Kubelet Volume Manager"
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: I1001 12:20:44.636135 30604 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: I1001 12:20:44.636728 30604 factory.go:223] Registration of the systemd container factory successfully
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: I1001 12:20:44.636945 30604 factory.go:221] Registration of the crio container factory failed: Get "http://%2Fvar%2Frun%2Fcrio%2Fcrio.sock/info": dial unix /var/run/crio/crio.sock: connect: no such file or directory
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: I1001 12:20:44.637656 30604 reconstruct.go:67] "Could not construct volume information" podName="c4c97b41-ba0a-4cc9-8479-02f099ce481b" volumeSpecName="pvc-322f7472-cecc-40cb-89f1-9b6c82c048a8" err="reconstructVolume.NewMounter failed for volume "kubernetes.io/local-volume/pvc-322f7472-cecc-40cb-89f1-9b6c82c048a8" (spec.Name: "pvc-322f7472-cecc-40cb-89f1-9b6c82c048a8") pod "c4c97b41-ba0a-4cc9-8479-02f099ce481b" (UID: "c4c97b41-ba0a-4cc9-8479-02f099ce481b") with: local volume source is nil or local path is not set"
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: E1001 12:20:44.638306 30604 kubelet.go:1600] "Image garbage collection failed once. Stats initialization may not have completed yet" err="invalid capacity 0 on image filesystem"
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: I1001 12:20:44.638332 30604 reconstruct.go:67] "Could not construct volume information" podName="f998cb64-9f06-41f5-9b11-0ef38a7c05bc" volumeSpecName="pvc-20f75c39-c925-4feb-8cbf-ffdfcf06b1cb" err="reconstructVolume.NewMounter failed for volume "kubernetes.io/local-volume/pvc-20f75c39-c925-4feb-8cbf-ffdfcf06b1cb" (spec.Name: "pvc-20f75c39-c925-4feb-8cbf-ffdfcf06b1cb") pod "f998cb64-9f06-41f5-9b11-0ef38a7c05bc" (UID: "f998cb64-9f06-41f5-9b11-0ef38a7c05bc") with: local volume source is nil or local path is not set"
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: I1001 12:20:44.638751 30604 reconstruct.go:67] "Could not construct volume information" podName="fb7257f0-f763-4eba-805a-064960c1d1fe" volumeSpecName="pvc-8abf7969-2aa6-4b6d-8c42-41f9ca81c9b3" err="reconstructVolume.NewMounter failed for volume "kubernetes.io/local-volume/pvc-8abf7969-2aa6-4b6d-8c42-41f9ca81c9b3" (spec.Name: "pvc-8abf7969-2aa6-4b6d-8c42-41f9ca81c9b3") pod "fb7257f0-f763-4eba-805a-064960c1d1fe" (UID: "fb7257f0-f763-4eba-805a-064960c1d1fe") with: local volume source is nil or local path is not set"
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: I1001 12:20:44.638891 30604 factory.go:223] Registration of the containerd container factory successfully
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: I1001 12:20:44.638976 30604 reconciler.go:26] "Reconciler: start to sync state"
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: time="2025-10-01T12:20:44-04:00" level=info msg="Updating embedded CRD addons.k3s.cattle.io"
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: I1001 12:20:44.656606 30604 cpu_manager.go:221] "Starting CPU manager" policy="none"
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: I1001 12:20:44.656825 30604 cpu_manager.go:222] "Reconciling" reconcilePeriod="10s"
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: I1001 12:20:44.656896 30604 state_mem.go:36] "Initialized new in-memory state store"
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: I1001 12:20:44.657091 30604 state_mem.go:88] "Updated default CPUSet" cpuSet=""
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: I1001 12:20:44.657148 30604 state_mem.go:96] "Updated CPUSet assignments" assignments={}
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: I1001 12:20:44.657223 30604 policy_none.go:49] "None policy: Start"
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: I1001 12:20:44.657272 30604 memory_manager.go:186] "Starting memorymanager" policy="None"
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: I1001 12:20:44.657321 30604 state_mem.go:35] "Initializing new in-memory state store"
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: I1001 12:20:44.657458 30604 state_mem.go:75] "Updated machine memory state"
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: E1001 12:20:44.663189 30604 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: I1001 12:20:44.663606 30604 eviction_manager.go:189] "Eviction manager: starting control loop"
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: I1001 12:20:44.663736 30604 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: I1001 12:20:44.663972 30604 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: E1001 12:20:44.664750 30604 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="no imagefs label for configured runtime"
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: time="2025-10-01T12:20:44-04:00" level=info msg="Updating embedded CRD etcdsnapshotfiles.k3s.cattle.io"
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: time="2025-10-01T12:20:44-04:00" level=info msg="Updating embedded CRD helmchartconfigs.helm.cattle.io"
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: I1001 12:20:44.708191 30604 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: I1001 12:20:44.710433 30604 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: I1001 12:20:44.710695 30604 status_manager.go:230] "Starting to sync pod status with apiserver"
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: I1001 12:20:44.710859 30604 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: I1001 12:20:44.710942 30604 kubelet.go:2436] "Starting kubelet main sync loop"
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: E1001 12:20:44.711052 30604 kubelet.go:2460] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: time="2025-10-01T12:20:44-04:00" level=info msg="Updating embedded CRD helmcharts.helm.cattle.io"
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: time="2025-10-01T12:20:44-04:00" level=warning msg="Failed to get apiserver addresses from supervisor: "
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: I1001 12:20:44.761922 30604 server.go:715] "Successfully retrieved node IP(s)" IPs=["{MY-OLD-SERVER-IP}"]
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: E1001 12:20:44.762172 30604 server.go:245] "Kube-proxy configuration may be incomplete or incorrect" err="nodePortAddresses is unset; NodePort connections will be accepted on all local IPs. Consider using--nodeport-addresses primary"
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: I1001 12:20:44.765058 30604 kubelet_node_status.go:75] "Attempting to register node" node="{SystemHostnameIG}"
Oct 01 12:20:44 {SystemHostnameIG} k3s[30604]: time="2025-10-01T12:20:44-04:00" level=fatal msg="Failed to start networking: unable to initialize network policy controller: error getting node subnet: failed to find interface with specified node ip"
Oct 01 12:20:44 {SystemHostnameIG} systemd[1]: k3s.service: Main process exited, code=exited, status=1/FAILURE
Oct 01 12:20:44 {SystemHostnameIG} systemd[1]: k3s.service: Failed with result 'exit-code'.
Oct 01 12:20:44 {SystemHostnameIG} systemd[1]: k3s.service: Consumed 19.240s CPU time.