support for control plane highly available

[pathcl]

Dear folks,

Im reading through garm codebase and already spotted there’s support for MySQL. Is it enough to configure garm as highly available control plane? my use case is on top of k8s.

[gabriel-samfira]

It used to have MySQL, but it became somewhat difficult to maintain, so we kind of dropped MySQL support.

But if you're trying to use it in k8s, I encourage you to use the garm-operator. The operator pretty much treats GARM as stateless and syncs the sqlite DB using the info it has stored in etcd.

The current push to move some things from the config to the DB is being done in order to eventually have GARM scale-out. So scaling out GARM is on the TODO list and we're working towards that, but even in the current state, it handles a large amount of runners with ease.

[pathcl]

Great! thanks for the quick reply. I tried the k8s operator but I understood it would also require a garm instance outside of the cluster or being reachable. Is this correct?

[gabriel-samfira]

You can have GARM run inside k8s without a problem. Have a look here:

https://github.com/mercedes-benz/garm-provider-k8s/blob/main/DEVELOPMENT.md

The instructions use tilt to bootstrap a local development environment along with garm, the operator and the k8s provider. You can use that as a starting point and expand to other providers you may need.

We need to add some proper docs in one place that gives a nice walk-through for the various cases.

[pathcl]

You can have GARM run inside k8s without a problem. Have a look here:

https://github.com/mercedes-benz/garm-provider-k8s/blob/main/DEVELOPMENT.md

The instructions use tilt to bootstrap a local development environment along with garm, the operator and the k8s provider. You can use that as a starting point and expand to other providers you may need.

We need to add some proper docs in one place that gives a nice walk-through for the various cases.

thanks for sharing that! would you say is the only thing and enough to start? I can improve docs once I get familiar with it

[gabriel-samfira]

That should bring up up and running with a fully functional GARM on k8s + operator. I usually run it as stand-alone, but I did manage to get it running using that guide.

@bavarianbidi may be able to chime in with more details. His wonderful team develops the k8s integration (operator and provider)

[pathcl]

That should bring up up and running with a fully functional GARM on k8s + operator. I usually run it as stand-alone, but I did manage to get it running using that guide.

@bavarianbidi may be able to chime in with more details. His wonderful team develops the k8s integration (operator and provider)

Are you using any specific commit? I can't get garm deployed.

garm-provider-k8s $ make tilt-up
hack/scripts/kind-with-registry.sh
No kind clusters found.
Creating cluster "garm" ...
 ✓ Ensuring node image (kindest/node:v1.28.7) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
Set kubectl context to "kind-garm"
You can now use your cluster with:

kubectl cluster-info --context kind-garm

Thanks for using kind! 😊
configmap/local-registry-hosting created
tilt up
Tilt started on http://localhost:10350/
v0.33.16, built 2024-06-07

(space) to open the browser
(s) to stream logs (--stream=true)
(t) to open legacy terminal mode (--legacy=true)
(ctrl-c) to exit

garm-provider-k8s $ git remote -v
origin	git@github.com:mercedes-benz/garm-provider-k8s (fetch)
origin	git@github.com:mercedes-benz/garm-provider-k8s (push)

garm-provider-k8s $ git rev-parse HEAD
b45a9889943b80d5d6e8222ab6c22a5f59e02157

garm-provider-k8s $ kubectl get pods -A
NAMESPACE            NAME                                         READY   STATUS    RESTARTS   AGE
kube-system          coredns-5dd5756b68-72csd                     1/1     Running   0          10s
kube-system          coredns-5dd5756b68-mmlqp                     1/1     Running   0          10s
kube-system          etcd-garm-control-plane                      1/1     Running   0          25s
kube-system          kindnet-5m6t5                                1/1     Running   0          11s
kube-system          kube-apiserver-garm-control-plane            1/1     Running   0          27s
kube-system          kube-controller-manager-garm-control-plane   1/1     Running   0          25s
kube-system          kube-proxy-r5ffl                             1/1     Running   0          11s
kube-system          kube-scheduler-garm-control-plane            1/1     Running   0          25s
local-path-storage   local-path-provisioner-7577fdbbfb-9q9ks      1/1     Running   0          10s

Garm should be deployed according to step 3) in https://github.com/mercedes-benz/garm-provider-k8s/blob/main/DEVELOPMENT.md#getting-started 🤔

[gabriel-samfira]

I used main but you may need to add description = "garm credentials" here:

https://github.com/mercedes-benz/garm-provider-k8s/blob/main/hack/local-development/kubernetes/configmap-envsubst.yaml#L47

But other than that, I just installed, docker, kubectl, tilt, go and went through the steps.

[gabriel-samfira]

you can also edit the existing config map:

kubectl -n garm-server edit configmap garm-configuration

and add it. Then remove the failing containers.

At the end you should have something like:

root@garm-deleteme:~# kubectl get pods -A
NAMESPACE              NAME                                                READY   STATUS    RESTARTS   AGE
cert-manager           cert-manager-5bd57786d4-jmwdj                       1/1     Running   0          58m
cert-manager           cert-manager-cainjector-57657d5754-89fwt            1/1     Running   0          58m
cert-manager           cert-manager-webhook-7d9f8748d4-npk9b               1/1     Running   0          58m
garm-operator-system   garm-operator-controller-manager-69fbd5c478-ctlqt   1/1     Running   0          47m
garm-server            garm-server-5b84b7f66-r7mxp                         1/1     Running   0          48m
kube-system            coredns-5dd5756b68-g8k87                            1/1     Running   0          58m
kube-system            coredns-5dd5756b68-wzxwj                            1/1     Running   0          58m
kube-system            etcd-garm-control-plane                             1/1     Running   0          59m
kube-system            kindnet-7r7mh                                       1/1     Running   0          58m
kube-system            kube-apiserver-garm-control-plane                   1/1     Running   0          59m
kube-system            kube-controller-manager-garm-control-plane          1/1     Running   0          59m
kube-system            kube-proxy-jz67s                                    1/1     Running   0          58m
kube-system            kube-scheduler-garm-control-plane                   1/1     Running   0          59m
local-path-storage     local-path-provisioner-7577fdbbfb-9bpx4             1/1     Running   0          58m

[pathcl]

you can also edit the existing config map:

kubectl -n garm-server edit configmap garm-configuration

and add it. Then remove the failing containers.

At the end you should have something like:

root@garm-deleteme:~# kubectl get pods -A
NAMESPACE              NAME                                                READY   STATUS    RESTARTS   AGE
cert-manager           cert-manager-5bd57786d4-jmwdj                       1/1     Running   0          58m
cert-manager           cert-manager-cainjector-57657d5754-89fwt            1/1     Running   0          58m
cert-manager           cert-manager-webhook-7d9f8748d4-npk9b               1/1     Running   0          58m
garm-operator-system   garm-operator-controller-manager-69fbd5c478-ctlqt   1/1     Running   0          47m
garm-server            garm-server-5b84b7f66-r7mxp                         1/1     Running   0          48m
kube-system            coredns-5dd5756b68-g8k87                            1/1     Running   0          58m
kube-system            coredns-5dd5756b68-wzxwj                            1/1     Running   0          58m
kube-system            etcd-garm-control-plane                             1/1     Running   0          59m
kube-system            kindnet-7r7mh                                       1/1     Running   0          58m
kube-system            kube-apiserver-garm-control-plane                   1/1     Running   0          59m
kube-system            kube-controller-manager-garm-control-plane          1/1     Running   0          59m
kube-system            kube-proxy-jz67s                                    1/1     Running   0          58m
kube-system            kube-scheduler-garm-control-plane                   1/1     Running   0          59m
local-path-storage     local-path-provisioner-7577fdbbfb-9bpx4             1/1     Running   0          58m

Thanks for the tip about configmap. I was able to fix that part. But now Im getting a different error:

$ kubectl get pool -n garm-operator-system -o yaml

  status:
    id: ""
    lastSyncError: referenced GitHubScopeRef Organization/my-org-here not ready yet
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

I created a PAT (classical token) but not sure what's going on. I followed this: https://github.com/mercedes-benz/garm-operator/blob/main/DEVELOPMENT.md#%EF%B8%8F-bootstrap-garm-server-with-garm-provider-k8s-for-local-development. Did you use a Github App for authentication?

[gabriel-samfira]

I used PAT auth. Make sure that the PAT you're using has access to the org/repo/enterprise you're creating and that you enabled the required scopes when creating the PAT. See:

https://github.com/cloudbase/garm/blob/main/doc/github_credentials.md

[gabriel-samfira]

ahh. I think I know what's happening. The operator is not yet updated to take into account the recent changes to GARM regarding the URLs. Try adding:

webhook_url = "http://garm-server.garm-server.svc:9997/webhooks"

here:

https://github.com/mercedes-benz/garm-provider-k8s/blob/b45a9889943b80d5d6e8222ab6c22a5f59e02157/hack/local-development/kubernetes/configmap-envsubst.yaml#L12

If you can connect using garm-cli to the garm server, you can also update using garm-cli controller update.

[gabriel-samfira]

I think it would be best if you switch garm to v0.1.4. The main branch has a bunch of updates and the operator has not caught up yet.

You can set v0.1.4 here:

https://github.com/mercedes-benz/garm-provider-k8s/blob/b45a9889943b80d5d6e8222ab6c22a5f59e02157/hack/Dockerfile#L6

[gabriel-samfira]

also, to get webhooks from GitHub, you'll most likely need an ingress controller and a cluster IP set on the GARM server. Then you'll need to add your webhook in GitHub to point to your GARM webhook URL.

See: https://github.com/cloudbase/garm/blob/v0.1.4/doc/webhooks.md

[pathcl]

also, to get webhooks from GitHub, you'll most likely need an ingress controller and a cluster IP set on the GARM server. Then you'll need to add your webhook in GitHub to point to your GARM webhook URL.

that's right but would I need a webhook to have a pool of runners working? I'm not sure.

BTW thanks to your help it worked! Im noticing these are configured as ephemeral runners by default:

garm-provider-k8s $ kubectl get pods -n runner
NAME                READY   STATUS      RESTARTS   AGE
garm-d7appm3tvuks   0/1     Completed   0          5m28s
garm-evctriwqbovn   0/1     Completed   0          5m28s

Do you think we could have Github Apps supported? it looks like it's already from garm-server side there but we're missing some bits between the release of v0.1.5 and garm-provider-k8s