Skip to content

Commit 18a9eb9

Browse files
kdodhia06kdodhia06
committed
Adding private cloud / On-prem clustered version
1 parent be7d746 commit 18a9eb9

File tree

6 files changed

+256
-0
lines changed

6 files changed

+256
-0
lines changed

‎clustered/.env‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
GIT_SHA=df2d92911aa0945f4079ce3b8f230c7fc13e1137
2+
STATIC_IP_LIST=172.18.0.2
3+
JITSU_SWAP_PATH=/tmp
4+
JITSU_SERVER_BUILD=Release
5+
WAREHOUSE_LOCATION=/tmp/jitsu-warehouse
6+
METASTORE_LOCATION=/tmp/jitsu-metastore
7+
JEC_CLUSTER_NAME=PureCompute
8+
JEC_MIN_ELASTIC_CLUSTER_NODES=0
9+
JEC_MAX_ELASTIC_CLUSTER_NODES=0
10+
MAX_SESSIONS_PER_NODE=2
11+
JEC_JITSU_SERVER_INSTANCE_TYPE=
12+
JEC_ORCHESTRATOR_INSTANCE_TYPE=
13+
JEC_JITSU_SERVER_INSTANCE_AMI=
14+
JEC_JITSU_SERVER_IAM_INSTANCE_PROFILE_NAME=
15+
JEC_JITSU_SERVER_IAM_INSTANCE_KEY_PAIR=
16+
JEC_SUBNET_ID=
17+
JEC_JITSU_SERVER_SECURITY_GROUP_ID=
18+
JEC_UNIQUE_TAG_VALUE=
19+
HIVE_SERVER2_IDLE_SESSION_TIMEOUT_SECONDS=3600
20+
HIVE_SERVER2_SESSION_CHECK_INTERVAL_SECONDS=100
21+
JEC_INSTANCE_CHECK_INTERVAL_SECONDS=3600000
22+
JDBC_PASSWORD=infinite-scale
23+
EFS_FILE_SYSTEM_ID=
24+
ENABLE_CACHING=false
25+
CACHE_CLUSTER_ID=
26+
CACHE_EXPIRY_TIME=
27+
MAX_CACHE_RESULT_SET_SIZE_MB=1
28+
JITSU_CUSTOMER_ID=
29+
JITSU_CUSTOMER_ATTRIBUTE=
30+
GET_SESSION_TIMEOUT_SECONDS=400
31+
GRPC_CONNECTION_MAX_RETRIES=300
32+
GRPC_CONNECTION_RETRY_SLEEP=1000

‎clustered/README.md‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
2+
# Private cloud / On-prem Cluster Deployment
3+
4+
## Overview
5+
6+
This guide will walk you through setting up and running a ComputeAI cluster. You'll need to allocate X nodes for Jitsu Servers and one node for the Orchestrator.
7+
8+
**Note:** This deployment is intended exclusively for private cloud or on-premises environments and should not be used in AWS, GCP, or Azure. For clustered deployments on AWS, GCP, or Azure, please reach out to the ComputeAI team to explore suitable product options.
9+
10+
## Prerequisites
11+
12+
- Docker and Docker Compose installed on each node.
13+
- Sufficient permissions to run Docker containers and modify environment variables.
14+
15+
## Setup Steps
16+
17+
1. **Clone or Transfer Setup Files:**
18+
19+
Ensure that each node has access to the setup files (`jitsuServer`, `orchestrator`, and `.env`). The directory structure should look like this on each node:
20+
21+
```
22+
/path/to/dir
23+
|__ .env
24+
├── config
25+
│ └── jitsuServerConfig.json
26+
├── jitsuServer
27+
│ └── docker-compose.yml
28+
└── orchestrator
29+
└── docker-compose.yml
30+
```
31+
32+
2. **Edit `.env` Configuration:**
33+
34+
Customize the `.env` file to match your specific cluster requirements. Some important parameters include:
35+
- `STATIC_IP_LIST`: A comma-separated list of IP addresses of the Jitsu Server nodes.
36+
- `JEC_CLUSTER_NAME`: The cluster name.
37+
- `JEC_WAREHOUSE_LOCATION` and `JEC_METASTORE_LOCATION`: Paths where warehouse and metastore data will be stored.
38+
39+
3. **Launch Jitsu Server Nodes:**
40+
41+
On each Jitsu Server node, perform the following steps:
42+
- Navigate to the `jitsuServer` directory.
43+
- Start the Jitsu Server using Docker Compose with the `.env` file:
44+
```bash
45+
cd /path/to/cluster/jitsuServer
46+
docker-compose --env-file ../.env up -d
47+
```
48+
- Confirm that the Jitsu Server is up and running on each node.
49+
50+
4. **Configure `STATIC_IP_LIST`:**
51+
52+
After all Jitsu Server nodes are up, obtain the IP addresses of the nodes and update the `STATIC_IP_LIST` variable in the `.env` file on the Orchestrator node. This will ensure that the Orchestrator knows which nodes to connect to.
53+
54+
Example:
55+
```env
56+
STATIC_IP_LIST=192.168.1.101,192.168.1.102,192.168.1.103
57+
```
58+
59+
5. **Start the Orchestrator:**
60+
61+
Once the `STATIC_IP_LIST` is updated, start the Orchestrator node:
62+
- Navigate to the `orchestrator` directory.
63+
- Start the Orchestrator service using Docker Compose with the `.env` file:
64+
```bash
65+
cd /path/to/cluster/orchestrator
66+
docker-compose --env-file ../.env up -d
67+
```
68+
- Verify that the Orchestrator service is up and managing the Jitsu Server nodes.
69+
70+
## Final Considerations
71+
72+
- **Logs and Monitoring:** Monitor the logs of each service by accessing the `logs` directory, or by viewing container logs via Docker.
73+
- **Security:** Ensure appropriate security measures are in place, especially for sensitive environment variables and network communications.
74+
- **Scaling:** Adjust the number of Jitsu Server nodes and resource allocation to match the expected workload.
75+

‎clustered/config/jitsuServerConfig.json‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
{
2+
"System": {
3+
"MaxSessionsPerSLACtx": 16,
4+
"LogLevel": "Info"
5+
},
6+
"Hypervisor": {
7+
"HypervisorMemoryInBytes": 104857600,
8+
"UseMaxCores": 1,
9+
"PageSz": 4096
10+
},
11+
"DefaultSLA": {
12+
"UseNormalFsAPIs": true,
13+
"PageBatchSz": 262144,
14+
"PageSz": 1048576
15+
}
16+
}

‎clustered/jitsuServer/docker-compose.yml‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
services:
2+
jitsuServer:
3+
image: jitsuai/compute_ai:jitsu-server-${GIT_SHA}
4+
container_name: jitsuServer
5+
restart: always
6+
ulimits:
7+
nofile: 1048576
8+
memlock: -1
9+
core: 0
10+
ports:
11+
- "50051:50051"
12+
expose:
13+
- 50051
14+
environment:
15+
- JITSU_SWAP_PATH=${JITSU_SWAP_PATH}
16+
- JITSU_SERVER_BUILD=${JITSU_SERVER_BUILD}
17+
- CONTAINER_NAME=jitsuServer
18+
- MaxSessionsPerSLACtx=${MAX_SESSIONS_PER_NODE}
19+
- TERMINFO=/usr/share/terminfo # To be able to run jitsu_top.py from within the container
20+
cap_add:
21+
- SYS_PTRACE
22+
security_opt:
23+
- seccomp=unconfined
24+
volumes:
25+
- ../config/jitsuServerConfig.json:/opt/jitsu/jitsuServer.json
26+
- /var/run/docker.sock:/var/run/docker.sock
27+
- /opt/jitsu/jitsu-warehouse:${WAREHOUSE_LOCATION}
28+
- /opt/jitsu/logs/:/opt/jitsu/logs/

‎clustered/orchestrator/docker-compose.yml‎

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
services:
2+
jupyternotebook:
3+
image: jitsuai/compute_ai:jupyter-notebook
4+
container_name: jupyter-notebook
5+
depends_on:
6+
- jitsuOrchestrator
7+
labels:
8+
description: Intel Python 2 using Jupyter Notebooks
9+
name: jupyter notebook
10+
expose:
11+
- 8888
12+
ports:
13+
- "8888:8888"
14+
volumes:
15+
- ~/notebooks:/home/notebooks
16+
command: jupyter notebook --NotebookApp.token='infinite-scale'
17+
18+
jitsuOrchestrator:
19+
image: jitsuai/compute_ai:jdbc-server-${GIT_SHA}
20+
container_name: jitsuOrchestrator
21+
environment:
22+
- GRPC_SERVER_PORT=50051
23+
- HIVE_SERVER2_IDLE_SESSION_TIMEOUT_SECONDS=${HIVE_SERVER2_IDLE_SESSION_TIMEOUT_SECONDS}
24+
- HIVE_SERVER2_SESSION_CHECK_INTERVAL_SECONDS=${HIVE_SERVER2_SESSION_CHECK_INTERVAL_SECONDS}
25+
- JEC_INSTANCE_CHECK_INTERVAL_SECONDS=${JEC_INSTANCE_CHECK_INTERVAL_SECONDS}
26+
- JDBC_AUTH=PASSWORD
27+
- CONTAINER_NAME=jitsuOrchestrator
28+
- JDBC_PORT=10000
29+
- DASHBOARD_PORT=8080
30+
- JDBC_PASSWORD=${JDBC_PASSWORD}
31+
- GRPC_CONNECTION_MAX_RETRIES=${GRPC_CONNECTION_MAX_RETRIES}
32+
- GRPC_CONNECTION_RETRY_SLEEP=${GRPC_CONNECTION_RETRY_SLEEP}
33+
- JEC_CLUSTER_NAME=${JEC_CLUSTER_NAME}
34+
- JEC_MIN_ELASTIC_CLUSTER_NODES=${JEC_MIN_ELASTIC_CLUSTER_NODES}
35+
- JEC_MAX_ELASTIC_CLUSTER_NODES=${JEC_MAX_ELASTIC_CLUSTER_NODES}
36+
- MAX_SESSIONS_PER_NODE=${MAX_SESSIONS_PER_NODE}
37+
- JEC_SUBNET_ID=${JEC_SUBNET_ID}
38+
- JEC_JITSU_SERVER_INSTANCE_TYPE=${JEC_JITSU_SERVER_INSTANCE_TYPE}
39+
- JEC_ORCHESTRATOR_INSTANCE_TYPE=${JEC_ORCHESTRATOR_INSTANCE_TYPE}
40+
- JEC_JITSU_SERVER_INSTANCE_AMI=${JEC_JITSU_SERVER_INSTANCE_AMI}
41+
- JEC_JITSU_SERVER_SECURITY_GROUP_ID=${JEC_JITSU_SERVER_SECURITY_GROUP_ID}
42+
- JEC_JITSU_SERVER_IAM_INSTANCE_PROFILE_NAME=${JEC_JITSU_SERVER_IAM_INSTANCE_PROFILE_NAME}
43+
- JEC_JITSU_SERVER_IAM_INSTANCE_KEY_PAIR=${JEC_JITSU_SERVER_IAM_INSTANCE_KEY_PAIR}
44+
- JEC_DEPLOYMENT=true
45+
- WAREHOUSE_LOCATION=${WAREHOUSE_LOCATION}
46+
- METASTORE_LOCATION=${METASTORE_LOCATION}
47+
- JEC_UNIQUE_TAG_VALUE=${JEC_UNIQUE_TAG_VALUE}
48+
- SPARK_LOG_DIR=/opt/jitsu/logs
49+
- EFS_FILE_SYSTEM_ID=${EFS_FILE_SYSTEM_ID}
50+
- ENABLE_CACHING=${ENABLE_CACHING}
51+
- CACHE_CLUSTER_ID=${CACHE_CLUSTER_ID}
52+
- CACHE_EXPIRY_TIME=${CACHE_EXPIRY_TIME}
53+
- MAX_CACHE_RESULT_SET_SIZE_MB=${MAX_CACHE_RESULT_SET_SIZE_MB}
54+
- JITSU_CUSTOMER_ID=${JITSU_CUSTOMER_ID}
55+
- JITSU_CUSTOMER_ATTRIBUTE=${JITSU_CUSTOMER_ATTRIBUTE}
56+
- PERCENT_MEMROY_TO_ALLOCATE_TO_SPARK_DRIVER=90
57+
- GET_SESSION_TIMEOUT_SECONDS=${GET_SESSION_TIMEOUT_SECONDS}
58+
- STATIC_IP_LIST=${STATIC_IP_LIST}
59+
expose:
60+
- 10000
61+
- 8080
62+
ports:
63+
- "10000:10000"
64+
- "8080:8080"
65+
volumes:
66+
- /var/run/docker.sock:/var/run/docker.sock
67+
- /opt/jitsu/jitsu-metastore:${METASTORE_LOCATION}
68+
- /opt/jitsu/jitsu-warehouse:${WAREHOUSE_LOCATION}
69+
- /opt/jitsu/logs/:/opt/jitsu/logs/

‎clustered/setup/ubuntu.sh‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
#!/bin/bash
2+
3+
# Update package list and install prerequisites
4+
sudo apt update
5+
sudo apt install -y apt-transport-https ca-certificates curl software-properties-common
6+
7+
# Add Docker’s official GPG key
8+
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
9+
10+
# Add Docker repository to APT sources
11+
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
12+
13+
# Update package list again with new Docker repository
14+
sudo apt update
15+
16+
# Install Docker Engine and Docker CLI
17+
sudo apt install -y docker-ce docker-ce-cli containerd.io
18+
19+
# Start Docker and enable it to start at boot
20+
sudo systemctl enable docker
21+
sudo systemctl start docker
22+
23+
# Add the current user to the Docker group
24+
sudo usermod -aG docker $USER
25+
26+
# Docker Compose installation (version 2.x as a Docker plugin)
27+
DOCKER_COMPOSE_VERSION="2.27.0"
28+
sudo curl -SL "https://github.com/docker/compose/releases/download/v${DOCKER_COMPOSE_VERSION}/docker-compose-linux-x86_64" -o /usr/local/bin/docker-compose
29+
sudo chmod +x /usr/local/bin/docker-compose
30+
31+
# Verify installations
32+
docker --version
33+
docker-compose --version
34+
35+
echo "Installation completed! Log out and log back in for Docker group changes to take effect."
36+

0 commit comments

Comments
 (0)