Skip to content

🌱 Change the experimental-e2e to use a two-node kind cluster #2382

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
openshift-merge-bot merged 1 commit into from
Dec 12, 2025
+55 −4
Merged

🌱 Change the experimental-e2e to use a two-node kind cluster #2382

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 10 additions & 4 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,10 @@ ifeq ($(origin KIND_CLUSTER_NAME), undefined)
KIND_CLUSTER_NAME := operator-controller
endif

ifeq ($(origin KIND_CONFIG), undefined)
KIND_CONFIG := ./kind-config.yaml
endif


ifneq (, $(shell command -v docker 2>/dev/null))
CONTAINER_RUNTIME := docker
Expand Down Expand Up @@ -263,7 +267,7 @@ image-registry: export GOARCH=amd64
image-registry: ## Build the testdata catalog used for e2e tests and push it to the image registry
go build $(GO_BUILD_FLAGS) $(GO_BUILD_EXTRA_FLAGS) -tags '$(GO_BUILD_TAGS)' -ldflags '$(GO_BUILD_LDFLAGS)' -gcflags '$(GO_BUILD_GCFLAGS)' -asmflags '$(GO_BUILD_ASMFLAGS)' -o ./testdata/push/bin/push ./testdata/push/push.go
$(CONTAINER_RUNTIME) build -f ./testdata/Dockerfile -t $(E2E_REGISTRY_IMAGE) ./testdata
$(CONTAINER_RUNTIME) save $(E2E_REGISTRY_IMAGE) | $(KIND) load image-archive /dev/stdin --name $(KIND_CLUSTER_NAME)
$(KIND) load docker-image $(E2E_REGISTRY_IMAGE) --name $(KIND_CLUSTER_NAME)
./testdata/build-test-registry.sh $(E2E_REGISTRY_NAMESPACE) $(E2E_REGISTRY_NAME) $(E2E_REGISTRY_IMAGE)

# When running the e2e suite, you can set the ARTIFACT_PATH variable to the absolute path
Expand All @@ -282,6 +286,7 @@ test-e2e: run-internal image-registry prometheus e2e e2e-coverage kind-clean #HE
.PHONY: test-experimental-e2e
test-experimental-e2e: SOURCE_MANIFEST := $(EXPERIMENTAL_E2E_MANIFEST)
test-experimental-e2e: KIND_CLUSTER_NAME := operator-controller-e2e
test-experimental-e2e: KIND_CONFIG := ./kind-config-experimental.yaml
test-experimental-e2e: GO_BUILD_EXTRA_FLAGS := -cover
test-experimental-e2e: COVERAGE_NAME := experimental-e2e
test-experimental-e2e: export MANIFEST := $(EXPERIMENTAL_RELEASE_MANIFEST)
Expand Down Expand Up @@ -382,8 +387,8 @@ stop-profiling: build-test-profiler #EXHELP Stop profiling and generate analysis

.PHONY: kind-load
kind-load: $(KIND) #EXHELP Loads the currently constructed images into the KIND cluster.
$(CONTAINER_RUNTIME) save $(OPCON_IMG) | $(KIND) load image-archive /dev/stdin --name $(KIND_CLUSTER_NAME)
$(CONTAINER_RUNTIME) save $(CATD_IMG) | $(KIND) load image-archive /dev/stdin --name $(KIND_CLUSTER_NAME)
$(KIND) load docker-image $(OPCON_IMG) --name $(KIND_CLUSTER_NAME)
$(KIND) load docker-image $(CATD_IMG) --name $(KIND_CLUSTER_NAME)

.PHONY: kind-deploy
kind-deploy: export DEFAULT_CATALOG := $(RELEASE_CATALOGS)
Expand All @@ -408,8 +413,9 @@ kind-deploy-experimental: manifests
.PHONY: kind-cluster
kind-cluster: $(KIND) kind-verify-versions #EXHELP Standup a kind cluster.
-$(KIND) delete cluster --name $(KIND_CLUSTER_NAME)
$(KIND) create cluster --name $(KIND_CLUSTER_NAME) --config ./kind-config.yaml
$(KIND) create cluster --name $(KIND_CLUSTER_NAME) --config $(KIND_CONFIG)
$(KIND) export kubeconfig --name $(KIND_CLUSTER_NAME)
kubectl wait --for=condition=Ready nodes --all --timeout=2m

.PHONY: kind-clean
kind-clean: $(KIND) #EXHELP Delete the kind cluster.
Expand Down
45 changes: 45 additions & 0 deletions kind-config-experimental.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
apiVersion: kind.x-k8s.io/v1alpha4
Copy link
Contributor

@pedjak pedjak Dec 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: kind-config-experimental.yaml sounds to me like some sort of experimental kind config, but actually it is just a two-node config. Perhaps we could reflect that in the filename?

BTW, does this file need to be in the project root?

Copy link
Contributor Author

@tmshort tmshort Dec 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's the same location as the other kind config; keep similar files together.

Copy link
Member

@joelanford joelanford Dec 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this file need to be in the project root

Now that there are multiple kind config files at play, it makes sense to me to bury this in a subdirectory somewhere.

Copy link
Contributor Author

@tmshort tmshort Dec 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's fair, and easy.

kind: Cluster
nodes:
- role: control-plane
extraPortMappings:
# e2e image registry service's NodePort
- containerPort: 30000
hostPort: 30000
listenAddress: "127.0.0.1"
protocol: tcp
# prometheus metrics service's NodePort
- containerPort: 30900
hostPort: 30900
listenAddress: "127.0.0.1"
protocol: tcp
kubeadmConfigPatches:
- |
kind: ClusterConfiguration
apiServer:
extraArgs:
enable-admission-plugins: OwnerReferencesPermissionEnforcement
- |
kind: InitConfiguration
nodeRegistration:
kubeletExtraArgs:
node-labels: "ingress-ready=true"
taints: []
extraMounts:
- hostPath: ./hack/kind-config/containerd/certs.d
containerPath: /etc/containerd/certs.d
- role: control-plane
Copy link
Contributor

@pedjak pedjak Dec 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we do not need another control-plane node, a worker node could be just fine. Furthermore, each control plane node runs a copy of control plane pods (etcd, apiserver, scheduler, controller-manager) doubling the resources consumed by kind. Additional worker node keeps the resource usage on previous level.

Copy link
Contributor Author

@tmshort tmshort Dec 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is on purpose,. operator-controller and catalogd are supposed to run only on control-plane nodes; not worker nodes. It's part of what we want to test. This is also only for the experimental-e2e, so it's not used for normal run-time.

Copy link
Contributor

@pedjak pedjak Dec 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we assert it during e2e tests on what nodes we run?

Copy link
Contributor Author

@tmshort tmshort Dec 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not yet, that was something that #2371 will be trying to do when they add 2 replicas.

kubeadmConfigPatches:
- |
kind: JoinConfiguration
nodeRegistration:
kubeletExtraArgs:
node-labels: "ingress-ready=true"
taints: []
extraMounts:
- hostPath: ./hack/kind-config/containerd/certs.d
containerPath: /etc/containerd/certs.d
containerdConfigPatches:
- |-
[plugins."io.containerd.grpc.v1.cri".registry]
config_path = "/etc/containerd/certs.d"