Skip to content
This repository was archived by the owner on May 14, 2025. It is now read-only.

Commit 915c16d

Browse files
author
Amit Kumar Das
authored
chore(examples): add dontpanic controller (#121)
This commit showcases how metac controller behaves when CRD is not applied against the k8s cluster but the controller tries to use an instance of this CRD. It is expected that the controller should not panic immediately & should provide root cause of the error. This is helpful since there can be cases where a CRD is missing but controllers try to use their instances without above knowledge. Signed-off-by: AmitKumarDas <amit.das@mayadata.io>
1 parent 732b0ee commit 915c16d

File tree

14 files changed

+942
-6
lines changed

14 files changed

+942
-6
lines changed

‎examples/gctl/dont-panic/.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
/vendor/

‎examples/gctl/dont-panic/Dockerfile‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
# --------------------------
2+
# Build d-operators binary
3+
# --------------------------
4+
FROM golang:1.13.5 as builder
5+
6+
WORKDIR /example.io/dontpanic/
7+
8+
# copy go modules manifests
9+
COPY go.mod go.mod
10+
COPY go.sum go.sum
11+
12+
COPY Makefile Makefile
13+
14+
COPY vendor/ vendor/
15+
16+
# ensure vendoring is up-to-date by running make vendor
17+
# in your local setup
18+
#
19+
# we cache the vendored dependencies before building and
20+
# copying source so that we don't need to re-download when
21+
# source changes don't invalidate our downloaded layer
22+
RUN go mod download
23+
RUN go mod tidy
24+
RUN go mod vendor
25+
26+
# copy source file(s)
27+
COPY cmd/ cmd/
28+
29+
# build the binary
30+
RUN make
31+
32+
# ---------------------------
33+
# Use distroless as minimal base image to package the final binary
34+
# Refer https://github.com/GoogleContainerTools/distroless
35+
# ---------------------------
36+
FROM gcr.io/distroless/static:nonroot
37+
38+
WORKDIR /
39+
40+
COPY --from=builder /example.io/dontpanic/dontpanic .
41+
COPY config/config.yaml /etc/config/metac/
42+
43+
USER nonroot:nonroot
44+
45+
CMD ["/dontpanic"]

‎examples/gctl/dont-panic/Makefile‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
ALL_SRC = $(shell find . -name "*.go" | grep -v -e "vendor")
2+
3+
IMG_NAME ?= dontpanic
4+
PACKAGE_VERSION ?= latest
5+
6+
all: bins
7+
8+
bins: vendor $(IMG_NAME)
9+
10+
$(IMG_NAME): $(ALL_SRC)
11+
@echo "+ Generating $(IMG_NAME) binary"
12+
@CGO_ENABLED=0 GOOS=linux GOARCH=amd64 GO111MODULE=on \
13+
go build -o $@ ./cmd/main.go
14+
15+
$(ALL_SRC): ;
16+
17+
# go mod download modules to local cache
18+
# make vendored copy of dependencies
19+
# install other go binaries for code generation
20+
.PHONY: vendor
21+
vendor: go.mod go.sum
22+
@GO111MODULE=on go mod download
23+
@GO111MODULE=on go mod tidy
24+
@GO111MODULE=on go mod vendor
25+
26+
.PHONY: image
27+
image:
28+
docker build -t $(IMG_NAME):$(PACKAGE_VERSION) .

‎examples/gctl/dont-panic/README.md‎

Lines changed: 118 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,118 @@
1+
## DontPanic kubernetes controller
2+
3+
This is an example of a k8s controller that demonstrates its availability irrespective of an runtime error or due to misconfigurations. In other words, controller should not crash.
4+
5+
- This is a k8s controller that imports metac as library
6+
- This enables use of inline hooks instead of webhooks
7+
- Controller's kubernetes resources are configured in a config file
8+
- Refer: config/config.yaml
9+
- Controller business logic is implemented in Go
10+
- Refer: cmd/main.go
11+
- Kubernetes client libraries are completely abstracted from this logic
12+
- Logic is implemented in respective reconcile functions
13+
- A CR `IAmError` is sent as the response i.e. desired state
14+
- NOTE: `IAmError`'s definition i.e. _(CRD)_ is not set in k8s cluster
15+
- Hence this reconciliation should **error out**
16+
- Expectations:
17+
- Binary should **not panic** inspite of above error
18+
- Log should provide the root cause of error
19+
- Docker image includes the binary as well as its config file
20+
- Refer: Dockerfile
21+
- Controller is deployed as a single StatefulSet
22+
- No need of separate metac binary since metac is imported as a library
23+
- Refer: dontpanic-operator.yaml
24+
25+
### Steps
26+
27+
```sh
28+
# workstation needs to have Docker
29+
# use kind to create a k8s cluster
30+
#
31+
# Refer: https://kind.sigs.k8s.io/docs/user/local-registry/
32+
sudo ./kind-with-registry.sh
33+
34+
# cat $HOME/.kube/config
35+
# connect to kind cluster
36+
sudo kubectl cluster-info --context kind-kind
37+
38+
# debugging info if required
39+
#
40+
# Kubernetes master is running at https://127.0.0.1:32774
41+
#
42+
# KubeDNS is running at:
43+
# https://127.0.0.1:32774/api/v1/namespaces/# kube-system/services/kube-dns:dns/proxy
44+
#
45+
# To further debug and diagnose cluster problems, use
46+
#'kubectl cluster-info dump'.
47+
```
48+
49+
```sh
50+
# NOTE:
51+
# - Docker daemon always runs as a root user
52+
# - sudo may not be required depending on individual confgurations
53+
# - sudo is needed if docker group is not configured
54+
# - KIND runs entirely as containers
55+
# - Hence, all kubectl commands might need to used with sudo
56+
```
57+
58+
```sh
59+
# workstation needs to have Docker
60+
make image
61+
62+
# tag the image to use the local registry
63+
sudo docker tag dontpanic:latest localhost:5000/dontpanic:latest
64+
65+
# push to local registry configured to be used by kind
66+
sudo docker push localhost:5000/dontpanic:latest
67+
```
68+
69+
```sh
70+
# install namespace, rbac, crds & operator
71+
sudo kubectl apply -f dontpanic-ns.yaml
72+
sudo kubectl apply -f dontpanic-rbac-crd.yaml
73+
sudo kubectl apply -f dontpanic-operator.yaml
74+
75+
# verify if above were installed properly
76+
sudo kubectl get ns
77+
sudo kubectl get crd
78+
sudo kubectl get sts -n dontpanic
79+
sudo kubectl describe po -n dontpanic
80+
sudo kubectl get po -n dontpanic
81+
sudo kubectl logs -n dontpanic dontpanic-0
82+
```
83+
84+
### Test
85+
86+
```sh
87+
# check operator pod
88+
sudo kubectl get pods -n dontpanic
89+
90+
# check operator pod logs
91+
sudo kubectl logs -n dontpanic dontpanic-0
92+
93+
# create the dontpanic custom resource
94+
sudo kubectl apply -f dontpanic.yaml
95+
96+
# check operator pod logs
97+
sudo kubectl logs -n dontpanic dontpanic-0
98+
```
99+
100+
### Observations
101+
- Binary did not panic
102+
- Following were the logs that points the root cause
103+
104+
```bash
105+
I0402 15:33:10.482481 1 discovery.go:174] API resources discovery completed
106+
I0402 15:33:10.651607 1 metacontroller.go:270] Condition failed: Will retry after 1s: Local GenericController: Failed to init dontpanic-controller: Local GenericController: Selector init failed: Can't find "iamerrors": Version "notsure.com/v1"
107+
```
108+
109+
### Cleanup
110+
111+
```sh
112+
sudo kubectl delete -f dontpanic.yaml
113+
sudo kubectl delete -f dontpanic-operator.yaml
114+
sudo kubectl delete -f dontpanic-rbac-crd.yaml
115+
sudo kubectl delete -f dontpanic-ns.yaml
116+
117+
sudo kind delete cluster
118+
```

‎examples/gctl/dont-panic/cmd/main.go‎

Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
package main
2+
3+
import (
4+
"github.com/golang/glog"
5+
"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
6+
"openebs.io/metac/controller/generic"
7+
"openebs.io/metac/start"
8+
)
9+
10+
// sync executes the reconciliation logic for the watched resource.
11+
// In this case resource under watch is a custom resource with kind
12+
// **DontPanic**.
13+
//
14+
// NOTE:
15+
// SyncHookRequest is the payload received as part of reconcile
16+
// request. Similarly, SyncHookResponse is the payload sent as a
17+
// response as part of reconcile request.
18+
//
19+
// NOTE:
20+
// SyncHookRequest has the resource that is under watch (here
21+
// DontPanic custom resource)
22+
//
23+
// NOTE:
24+
// Both SyncHookRequest & SyncHookResponse have the resources that
25+
// form the desired state based on the watched resource's specs.
26+
// These desired resources are termed as attachments by metac.
27+
// SyncHookRequest has these attachments filled up by metac based
28+
// on what is observed in the k8s cluster. SyncHookResponse's
29+
// attachments needs to be filled up with what is desired by this
30+
// controller logic.
31+
func sync(
32+
request *generic.SyncHookRequest,
33+
response *generic.SyncHookResponse,
34+
) error {
35+
glog.Infof("Starting DontPanic sync")
36+
defer glog.Infof("Completed DontPanic sync")
37+
38+
// reconciliation returns a desired CR whose definition
39+
// i.e. CRD is not applied to the k8s cluster
40+
//
41+
// In other words, k8s cluster does not understand
42+
// IAmError kind
43+
desired := &unstructured.Unstructured{
44+
Object: map[string]interface{}{
45+
"kind": "IAmError",
46+
"apiVersion": "notsure.com/v1",
47+
"metadata": map[string]interface{}{
48+
"name": request.Watch.GetName(),
49+
"namespace": request.Watch.GetNamespace(),
50+
"annotations": map[string]interface{}{
51+
// this annotation helps in associating the watch
52+
// with its corresponding attachment
53+
"dontpanic/uid": string(request.Watch.GetUID()),
54+
},
55+
},
56+
"spec": map[string]interface{}{
57+
"message": "My CRD is not installed!",
58+
},
59+
},
60+
}
61+
// add this desired instance to response to let metac create
62+
// it in the k8s cluster
63+
response.Attachments = append(
64+
response.Attachments,
65+
desired,
66+
)
67+
return nil
68+
}
69+
70+
// finalize will delete the attachment(s) created during
71+
// reconciliation of the watch
72+
//
73+
// NOTE:
74+
// Presence of finalize in the config automatically adds
75+
// finalizers against the watch
76+
//
77+
// NOTE:
78+
// Once attachments are deleted from the cluster,
79+
// 'response.Finalized' is set to true which in turn removes
80+
// this finalizers from the watch
81+
func finalize(
82+
request *generic.SyncHookRequest,
83+
response *generic.SyncHookResponse,
84+
) error {
85+
glog.Infof("Starting DontPanic finalize")
86+
defer glog.Infof("Completed DontPanic finalize")
87+
88+
if request.Attachments.IsEmpty() {
89+
response.Finalized = true
90+
}
91+
return nil
92+
}
93+
94+
func main() {
95+
generic.AddToInlineRegistry("sync/dontpanic", sync)
96+
generic.AddToInlineRegistry("finalize/dontpanic", finalize)
97+
start.Start()
98+
}

‎examples/gctl/dont-panic/config/config.yaml‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
apiVersion: metac.openebs.io/v1alpha1
2+
kind: GenericController
3+
metadata:
4+
name: dontpanic-controller
5+
spec:
6+
watch:
7+
apiVersion: example.com/v1
8+
resource: dontpanics
9+
attachments:
10+
- apiVersion: notsure.com/v1
11+
resource: iamerrors
12+
advancedSelector:
13+
selectorTerms:
14+
- matchReferenceExpressions:
15+
# select IAmError if its annotation
16+
#
17+
# matches DontPanic _(i.e. watch)_ UID
18+
- key: metadata.annotations.dontpanic/uid
19+
operator: EqualsWatchUID
20+
hooks:
21+
sync:
22+
inline:
23+
funcName: sync/dontpanic
24+
finalize:
25+
inline:
26+
funcName: finalize/dontpanic

‎examples/gctl/dont-panic/dontpanic-ns.yaml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
apiVersion: v1
2+
kind: Namespace
3+
metadata:
4+
name: dontpanic

‎examples/gctl/dont-panic/dontpanic-operator.yaml‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
---
2+
apiVersion: apps/v1
3+
kind: StatefulSet
4+
metadata:
5+
labels:
6+
app.mayadata.io/name: dontpanic
7+
name: dontpanic
8+
namespace: dontpanic
9+
spec:
10+
replicas: 1
11+
serviceName: ""
12+
selector:
13+
matchLabels:
14+
app.mayadata.io/name: dontpanic
15+
template:
16+
metadata:
17+
labels:
18+
app.mayadata.io/name: dontpanic
19+
spec:
20+
serviceAccountName: dontpanic
21+
containers:
22+
- name: dontpanic
23+
image: localhost:5000/dontpanic:latest # local registry
24+
command: ["/dontpanic"]
25+
args:
26+
- --logtostderr
27+
- --run-as-local
28+
- --workers-count=1 # number of workers per controller
29+
- --discovery-interval=40s
30+
- --cache-flush-interval=240s # re-sync interval
31+
- -v=7 # this will give us startup errors if any
32+
---

0 commit comments

Comments
 (0)