Contour development locally on macOS using Colima

[tsaarni]

I'm mainly a Linux user but sometimes I develop on mac as well. Since Docker Desktop is commercial software, I would prefer colima instead. If you have not heard of it, Colima is a tool for running container runtimes on mac. By default it uses the free Docker Engine runtime, also known as "Community Edition". I'll include some tips here on how I configured it for local development of Contour itself. I prefer fast feedback cycle (who wouldn't?) so I usually try to avoid container image builds & re-deployment, and sometimes I need to step through the code under debugger.

As a pre-condition, I assume you already have git, golang, vscode (or other IDE) and that you have cloned Contour source code to your machine.

Install following tools

brew install colima kubectl

To start colima run

colima start --kubernetes --k3s-arg "--disable=traefik --disable=servicelb"

This will launch new Ubuntu VM with Kubernetes in it, k3s to be more precise. The additional arguments will avoid installing the default ingress controller and load balancer, since we intend to run our own.

Next deploy Contour

kubectl apply -f https://projectcontour.io/quickstart/contour.yaml

To test that everything works, deploy echoserver (check the manifest here)

kubectl apply -f https://gist.githubusercontent.com/tsaarni/eb445b28e71cc64ed1a7f0c21e5ecced/raw/echoserver.yaml

Wait for the pod to come up and make a HTTP request

curl http://echoserver.127-0-0-1.nip.io

This will work directly from mac because Colima has set up port forwarding from localhost to the VM. Envoy container uses hostports to bind to ports 80 and 443. For having real hostname for the virtualhost, I used nip.io.

You can (re)build and deploy your own Contour container image, and for certain type of testing it can be better approach. On other cases, I run Contour locally on mac. For that to work, we need Envoy to connect to a process outside the Kubernetes cluster which requires few tricks but it is not that complicated at the end. The required information is found in the lima documentation: "The loopback addresses of the host is 192.168.5.2". So we need to instruct Envoy to connect to 192.168.5.2.

First shut down Contour pod by scaling down the instances

kubectl -n projectcontour scale deployment --replicas=0 contour

Envoy will connect to hostname contour to receive its configuration. One approach is to change the XDS server address in Envoy bootstrap configuration file, but I've used an alternative way: I "detach" the Service from the Contour pod (which is not anymore running) and manually create Endpoints object that points contour to 192.168.5.2:

cat <<EOF | kubectl apply -f -
kind: Service
apiVersion: v1
metadata:
  name: contour
  namespace: projectcontour
spec:
  type: ClusterIP
  ports:
  - port: 8001
    targetPort: 8001
---
kind: Endpoints
apiVersion: v1
metadata:
  name: contour
  namespace: projectcontour
subsets:
 - addresses:
     - ip: 192.168.5.2
   ports:
     - port: 8001
EOF

You can restart Envoy to have it immediately connect to the new address

kubectl -n projectcontour rollout restart daemonset envoy

Envoy uses TLS to connect to Contour so we need Contour's server certificate and private key, and CA certificate available locally. Run following on Contour source directory to download them from the cluster

kubectl -n projectcontour get secret contourcert -o jsonpath='{..ca\.crt}' | base64 -d > ca.crt
kubectl -n projectcontour get secret contourcert -o jsonpath='{..tls\.crt}' | base64 -d > tls.crt
kubectl -n projectcontour get secret contourcert -o jsonpath='{..tls\.key}' | base64 -d > tls.key

Configure vscode debugger for launching contour serve

mkdir -p .vscode
cat <<EOF > .vscode/launch.json
{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "Run contour",
      "type": "go",
      "request": "launch",
      "mode": "auto",
      "cwd": "\${workspaceRoot}",
      "program": "cmd/contour",
      "args": [
        "serve",
        "--xds-address=0.0.0.0",
        "--xds-port=8001",
        "--envoy-service-http-port=8080",
        "--envoy-service-https-port=8443",
        "--contour-cafile=ca.crt",
        "--contour-cert-file=tls.crt",
        "--contour-key-file=tls.key",
        "--debug"
      ]
    }
  ]
}
EOF

Now we are ready to run Contour under debugger and enjoy super fast development!

Note that Contour will be able to connect to Kubernetes API server without any additional configuration because Colima has set up Kubernetes admin credentials and server address to ~/.kube/config. The cluster admin is more powerful than the service account that Contour uses when running inside cluster. Be aware of this when testing functionality related to RBAC! It is possible to copy service account with correct privileges from the cluster but for most cases it is not really necessary.

Happy hacking!