Sample Kubernetes project for Multicloud

Infrastructure as Code (IaC) for AWS, Google Kubernetes Engine (GKE) or Minikube using Terraform/Helm

This project demonstrates an implementation of Infrastructure as Code (IaC) for deploying various architectures such as AWS, Google Kubernetes Engine (GKE) or Minikube using Terraform/Helm. Please keep in mind that this is a sample project and is not suitable for production environments.

Prerequisites

Before getting started, please ensure that you have the following software installed:

• Task: To simplify the command line interface.
• Pre-commit: To run various linting and formatting tools.
• kubectl
• Helm
• Terraform
• Minikube
• Docker
• AWS CLI
• Google Cloud CLI

With this project, you can find

This project includes the following features:

• AWS
  • Bedrock Agent Knowledge Base
  • EC2 Stack
• GCP
  • Google Kubernetes Engine (GKE)
  • Kubernetes Nginx Ingress Controller
  • Kong Ingress Controller - DB-less
  • Cert Manager
  • MySQL InnoDBCluster
  • Prometheus/Grafana
• Minikube
  • Kubernetes Nginx Ingress Controller
  • Kong Ingress Controller - DB-less
  • Cert Manager
  • MySQL InnoDBCluster
  • Prometheus/Grafana
• Helm charts for deploying multiple containerized microservices
• (WIP) CI/CD to build/lint/deploy infrastructure:
  • Gitlab
  • Github Actions
• Various bash scripts to deploy the cluster

Task commands

task setup:repo
task docker:exec
$ task precommit

Provision AWS Bedrock Agent Knowledge Base

To provision AWS Bedrock Agent Knowledge Base, follow these steps:

task docker:exec

task aws:bedrockagent:terraform:init
task aws:bedrockagent:terraform:workspace:list
task aws:bedrockagent:terraform:workspace:select
task aws:bedrockagent:terraform:workspace:new # if required
task aws:bedrockagent:terraform:validate
task aws:bedrockagent:terraform:plan
task aws:bedrockagent:terraform:apply
task aws:bedrockagent:terraform:destroy

Provision AWS EC2 Stack

To provision AWS EC2 Stack, follow these steps:

task docker:exec

task aws:ec2-stack:terraform:init
task aws:ec2-stack:terraform:workspace:list
task aws:ec2-stack:terraform:workspace:select
task aws:ec2-stack:terraform:workspace:new # if required
task aws:ec2-stack:terraform:validate
task aws:ec2-stack:terraform:plan
task aws:ec2-stack:terraform:apply
task aws:ec2-stack:terraform:destroy

Provision a Kubernetes cluster with Google Kubernetes Engine (GKE)

To provision a cluster with GKE, follow these steps:

1. Launch the orchestrator by running the following command:

   task docker:exec

2. Make the orchestrator accessible to the GKE cluster by running the following command: Note that the orchestrator should've authenticate when launching the bash shell.

   gcloud auth list

3. Run the Terraform commands by navigating to the appropriate directory and running the following commands:

   task gcp:gke-stack:terraform:init
   task gcp:gke-stack:terraform:workspace:list
   task gcp:gke-stack:terraform:workspace:select
   task gcp:gke-stack:terraform:workspace:new # if required
   task gcp:gke-stack:terraform:validate
   task gcp:gke-stack:terraform:plan
   task gcp:gke-stack:terraform:apply
   task gcp:gke-stack:terraform:destroy

   After applying Terraform, it will output the load balancer IP address.

   load_balancer_ip_address = "12.123.123.12"

4. Add an A record to your domain’s DNS records. In this repo, it was nvm.chrislee.kr.

Retrieving the Kubernetes Context

To retrieve the Kubernetes context, run the following commands:

gcloud projects list
gcloud container clusters list
gcloud container clusters get-credentials <cluster name> --region australia-southeast2 --project <project id>

Provision with Minikube

To provision a cluster with Minikube, follow these steps:

1. Start Minikube by running the following command:

   minikube start --addons metrics-server

   Wait until the Minikube cluster is provisioned.

   And open another terminal to make a tunnel for Load Balancer

   sudo minikube tunnel

   If wants nginx ingress controller, then enable Minikube ingress.

   minikube addons enable ingress

2. Launch the orchestrator by running the following command:

   task docker:exec
   
   # install docker
   apk add --no-cache docker

3. Make the orchestrator accessible to the Minikube cluster by running the following script:

   /srv/workspaces/minikube/scripts/set-kube-context.sh

4. Run the Terraform commands by navigating to the appropriate directory and running the following commands:

   task local:minikube-stack:terraform:init
   task local:minikube-stack:terraform:workspace:list
   task local:minikube-stack:terraform:workspace:select
   task local:minikube-stack:terraform:workspace:new # if required
   task local:minikube-stack:terraform:validate
   task local:minikube-stack:terraform:plan
   task local:minikube-stack:terraform:apply
   task local:minikube-stack:terraform:destroy

5. Update your host file with the following entry:

   vim /etc/hosts
   127.0.0.1   nvm-boilerplate.local

6. Open a new browser and navigate to nvm-boilerplate.local

Microservices Repository: Node.js + Vue.js + MySQL Boilerplate

For more information about the microservices used in this project, please visit https://github.com/chrisleekr/nodejs-vuejs-mysql-boilerplate

Accessing MySQL with Oracle MySQL Operator

To access MySQL, run the following commands:

# Get root password
$ kubectl -nmysql get secrets mysql-innodbcluster-cluster-secret -oyaml
$ echo "<rootPassword>" | base64 -d

# Port forward
$ kubectl -nmysql port-forward svc/mysql-innodbcluster 6446:6446

# Access to R/W MySQL
$ mysql -h127.0.0.1 -uroot -p -P6446 boilerplate

Using Horizontal Pod Autoscaler

To view information about Horizontal Pod Autoscaler, run the following command:

kubectl get hpa --all-namespaces

If you see <unknown>/50% when using Minikube, make sure you have enabled metrics-server by running this command:

minikube addons enable metrics-server

Monitoring with Prometheus & Grafana

You can access Grafana via http://nvm-boilerplate.local/grafana when using Minikube.

After the deployment is completed, you will see output similar to the following:

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

Outputs:

grafana_admin_password = <sensitive>
mysql_boilerplate_password = <sensitive>
mysql_root_password = <sensitive>

You can retrieve the Grafana admin password by running the following command:

terraform output grafana_admin_password

With the password, you can log in to Grafana using admin/<Password>.

In addition, you can access Prometheus http://nvm-boilerplate.local/prometheus in Minikube.

Todo

• Update MySQL with a replicated stateful application - Use presslabs/mysql-operator
• Add HorizontalPodAutoscaler
• Add Prometheus and Grafana
• Expose MySQL write node for migration to avoid api migration failure
• Replaced presslab/mysql-operator to Oracle MySQL operator/InnoDB cluster
• Support Google Kubernetes Engine
• Support Kong ingress controller