clusterctl move breaks control-plane load balancing #723
Description
The PR #233 moved the DO LB tag from ClusterNameRoleTag to ClusterNameUIDRoleTag.
This was necessary to have a differentiation between clusters in different namespaces and the cluster UID makes this unique.
However k8s UIDs cannot be migrated from cluster to cluster, they're generated by kube-apiserver when the object is created.
Clusterctl move moves all kubernetes objects which causes them to get new UIDs and then uses a mechanism to re-create the owner refs between those objects to point to the right UIDs.
This leaves behind a load-balancer that has the old UID tag and does service discover through that. This works as long as the control-plane machines aren't recycled because these also still have the old tag. Once they get replaced with new machines, those new machines will have the new cluster UID in the tag and therefore not match the LB tag based discovery anymore. The result is an offline kube control-plane.
It's possible to recover this manually by fixing the tag in the LB, but we should find a way to make that moving process smoother and fully automatic.