ZenML Infrastructure] + B -->|2. Auth Token| A + A -->|3. Access Workspace| C[Workspace
Your Infrastructure] + C -->|4. Validate Token| B + B -->|5. Authorization| C + C -->|6. Execute| D[Your Resources] +``` + +1. User authenticates with ZenML control plane (SSO) +2. Control plane issues authentication credentials +3. User accesses workspace with credentials +4. Workspace validates credentials with control plane +5. Control plane confirms authenticaiton and authorization (RBAC) +6. Workspace executes operations on your infrastructure + +### Data Residency + +| Data Type | Storage Location | Purpose | +|-----------|-----------------|---------| +| User metadata | Control Plane | Authentication only | +| RBAC policies | Control Plane | Authorization decisions | +| Pipeline metadata | Your Infrastructure | Run history, metrics, parameters | +| Model metadata | Your Infrastructure | Model versions, stages, annotations | +| Artifacts | Your Infrastructure | Datasets, models, visualizations | +| Secrets | Your Infrastructure | Cloud credentials, API keys | +| Logs | Your Infrastructure | Step outputs, debug information | + +## Deployment Architecture + +### Single Organization, Multiple Workspaces + +```mermaid +graph TB + subgraph clients["Client Machines (Developer Laptops/CI)"] + C1[Data Scientist] + C2[ML Engineer] + C3[CI/CD Pipeline] + end + + subgraph zenml["ZenML Infrastructure"] + CP[Control Plane
- Authentication SSO
- RBAC Management
- Workspace Registry] + end + + subgraph customer["Your Infrastructure"] + subgraph ws1["Workspace 1 - Team A"] + W1[ZenML Server
Metadata DB
Secrets Store] + R1[Your Resources
Orchestrator
Artifact Store] + end + + subgraph ws2["Workspace 2 - Team B"] + W2[ZenML Server
Metadata DB
Secrets Store] + R2[Your Resources
Orchestrator
Artifact Store] + end + + subgraph wsn["Workspace N - Platform"] + WN[ZenML Server
Metadata DB
Secrets Store] + RN[Your Resources
Orchestrator
Artifact Store] + end + end + + C1 -->|1. Authenticate| CP + C2 -->|1. Authenticate| CP + C3 -->|1. Authenticate| CP + + CP -->|2. RBAC Token| C1 + CP -->|2. RBAC Token| C2 + CP -->|2. RBAC Token| C3 + + C1 -->|3. Run Pipeline| W1 + C2 -->|3. Run Pipeline| W2 + C3 -->|3. Run Pipeline| WN + + W1 -.->|Validate Token| CP + W2 -.->|Validate Token| CP + WN -.->|Validate Token| CP + + W1 -->|Execute| R1 + W2 -->|Execute| R2 + WN -->|Execute| RN + + style zenml fill:#e1f5ff + style customer fill:#f0f0f0 + style clients fill:#fff4e6 +``` + +**Connection Flow:** +1. **Clients authenticate** with ZenML Control Plane (SSO) - hosted by ZenML +2. **Control Plane issues** RBAC-validated tokens to clients +3. **Clients connect** to their assigned workspace(s) in your infrastructure +4. **Workspaces validate** tokens with Control Plane (outbound-only connection) +5. **Pipelines execute** on your infrastructure resources + +### Multi-Region Support + +Deploy workspaces across different regions while maintaining centralized control: +- Workspaces in US, EU, APAC regions +- Data residency compliance per region +- Centralized user management +- Consistent RBAC across regions + +## Setup Process + +### 1. Initial Configuration + +[Book a demo](https://www.zenml.io/book-your-demo) to get started. The ZenML team will: +- Help set up your organization in the control plane +- Establish secure communication channels +- (optional) Configure SSO integration + +### 2. Workspace Deployment + +Deploy ZenML workspaces in your infrastructure. Workspaces can be deployed on: + +**Supported Deployment Backends:** +- **Kubernetes** (Recommended) - EKS, GKE, AKS, or self-managed clusters +- **AWS ECS** - Elastic Container Service +- **Container orchestration alternatives** - Other Kubernetes distributions + +**Requirements:** +- **Database**: MySQL or PostgreSQL database in your infrastructure +- **Network**: Egress access to `cloud.zenml.io` (for Control Plane communication) +- **Resources**: Compute resources for the ZenML server container + +**Deployment Tools:** +- **Kubernetes**: We provide officially supported Helm charts +- **Non-Kubernetes environments**: We recommend using infrastructure-as-code tools like Terraform, Pulumi, or CloudFormation to manage server lifecycle + + +### 3. Configure Infrastructure Access + +Once your workspace is deployed, configure access to your cloud resources using ZenML's infrastructure abstractions: + +**Stack Components**: Individual infrastructure elements that your pipelines need to run - orchestrators (Kubernetes, Airflow, etc.), artifact stores (S3, GCS, Azure Blob), container registries, experiment trackers, model deployers, and more. Each component type has multiple "flavors" supporting different technologies. + +**Stacks**: A stack is a named collection of components that define where and how your pipelines run. By combining different components into stacks, you can easily switch between environments (development, staging, production) or infrastructure providers without changing your pipeline code. + +**Service Connectors**: Service connectors provide secure, reusable authentication to cloud providers and services. Instead of managing credentials manually in each component, connectors handle authentication centrally and can be shared across your team with appropriate access controls. + +Learn more: +- [Stack Components Documentation](https://docs.zenml.io/stacks) - Available components and how to configure them +- [Stacks Documentation](https://docs.zenml.io/user-guide/production-guide/understand-stacks) - Complete guide to configuring and managing stacks +- [Service Connectors Documentation](https://docs.zenml.io/how-to/auth-management/service-connectors-guide) - How to set up authentication to cloud providers + +### 4. Set Up Users & Teams + +Manage users through the control plane: +- Invite team members via email +- Assign roles and permissions +- Create teams for different departments +- Configure workspace access + + +## Organizational Structure + +### Recommended Hierarchy + +```mermaid +graph TB + subgraph cp["Control Plane (ZenML Infrastructure)"] + ORG[Organization] + PT[Platform Team
Org Admins] + end + + subgraph infra["Your Infrastructure"] + subgraph ws1["DS Team 1 Workspace"] + W1[ZenML Server
Metadata DB] + T1[Team Members] + S1[Stacks
managed by Platform Team] + end + + subgraph ws2["DS Team 2 Workspace"] + W2[ZenML Server
Metadata DB] + T2[Team Members] + S2[Stacks
managed by Platform Team] + end + end + + ORG --> PT + PT -.->|Configure & Manage| S1 + PT -.->|Configure & Manage| S2 + PT -.->|Cross-workspace
Admin Access| W1 + PT -.->|Cross-workspace
Admin Access| W2 + + T1 -->|Use Stacks
Run Pipelines
Create Projects| W1 + T2 -->|Use Stacks
Run Pipelines
Create Projects| W2 + + style cp fill:#e1f5ff + style infra fill:#f0f0f0 + style PT fill:#ffd700 + style T1 fill:#98fb98 + style T2 fill:#98fb98 +``` + +**Access Model:** +- **Platform Team**: Organization admins with cross-workspace access. They configure and manage stacks, service connectors, and infrastructure across all workspaces +- **DS/ML Teams**: Limited workspace-level access. Can use pre-configured stacks to run pipelines, create projects, and manage workspace-level secrets, but cannot modify stack configurations or global settings +- **Workspace Isolation**: Each workspace runs independently in your infrastructure with its own ZenML server and metadata store + +## Cost Considerations + +### Infrastructure Costs +You control costs by managing: +- Compute resources (scale up/down as needed) +- Storage (artifact stores, databases) +- Networking (data transfer, load balancers) +- Backups and disaster recovery + +### ZenML Costs +ZenML provides: +- Control plane management (included) +- Professional support (included) +- Regular updates and security patches +- Usage-based pricing per workspace + +## Security Documentation + +For software deployed on your infrastructure, ZenML provides: + +- **Vulnerability Assessment Reports**: Comprehensive security analysis available on request +- **Software Bill of Materials (SBOM)**: Complete dependency inventory for compliance +- **Compliance documentation**: Support for your security audits and certifications +- **Architecture review**: Security team consultation for deployment planning + +Contact [cloud@zenml.io](mailto:cloud@zenml.io) to request security documentation. + +## Monitoring & Maintenance + +### Control Plane (ZenML Managed) +- ✅ Automatic updates +- ✅ Security patches +- ✅ Uptime monitoring +- ✅ Backup and recovery + +### Workspaces (Your Responsibility) +- Database maintenance and backups +- Workspace version updates (with ZenML guidance) +- Infrastructure scaling +- Resource monitoring + +### Support Included +- Professional support with SLA +- Architecture consultation +- Migration assistance +- Security advisory updates + +## Comparison with Other Deployments + +| Feature | SaaS | Hybrid SaaS | Self-hosted | +|---------|------|-------------|------------| +| Setup Time | Minutes | Hours to Days | Days to Weeks | +| Metadata Location | ZenML Infra | Your Infra | Your Infra | +| Secret Management | ZenML or Yours | Your Infra | Your Infra | +| User Management | ZenML Managed | ZenML Managed | Self-Managed | +| Maintenance | Zero | Workspace Only | Full Stack | +| Control | Minimal | Moderate | Complete | +| Best For | Fast start | Security + Convenience | Strictest compliance | + +[Compare all deployment options →](README.md#deployment-scenarios) + +## Migration Paths + +### From ZenML OSS +1. Deploy a ZenML Pro-compatible workspace in your own infrastructure (you can start from your existing ZenML OSS workspace deployment). + - **Update your Docker image**: Replace the OSS ZenML server image with the latest Pro Hybrid image provided by ZenML. + - **Set required environment variables**: Add or update environment variables according to the ZenML Pro documentation (for example: `ZENML_PRO_CONTROL_PLANE_URL`, `ZENML_PRO_CONTROL_PLANE_CLIENT_ID`, secrets, and SSO configuration as instructed by ZenML). + - **Restart your deployment** to apply these changes. +2. Migrate users and teams +5. Run `zenml login` to authenticate via [cloud.zenml.io](https://cloud.zenml.io) and connect your SDK clients to the new workspace + +### From SaaS to Hybrid + +If you're interested in migrating from the ZenML Pro SaaS deployment to a Hybrid SaaS setup, we're here to help guide you through every step of the process. Because migration paths can vary depending on your organization’s size, data residency requirements, and current ZenML setup, we recommend discussing your plans with a ZenML solutions architect. + +**Next steps:** + +- [Book a migration consultation →](https://www.zenml.io/book-your-demo) +- Or email us at [cloud@zenml.io](mailto:cloud@zenml.io) + +Your ZenML representative will provide you with a tailored migration checklist, technical documentation, and direct support to ensure a smooth transition with minimal downtime. + + +### Between Workspaces + +A workspace deep copy feature for migrating pipelines and artifacts between workspaces is coming soon. + +## Detailed Architecture Diagram + + + +## Related Resources + +- [System Architecture Overview](../system-architectures.md#zenml-pro-hybrid-saas) +- [Deployment Scenarios Overview](deployments-overview.md) +- [SaaS Deployment](saas-deployment.md) +- [Self-hosted Deployment](self-hosted-deployment.md) +- [Workload Managers](workload-managers.md) +- [Self-hosted Deployment Guide](self-hosted.md) +- [Workspaces](workspaces.md) +- [Organizations](organization.md) + +## Get Started + +Ready to deploy ZenML Pro in Hybrid mode? + +[Book a Demo](https://www.zenml.io/book-your-demo){ .md-button .md-button--primary } + +Have questions? [Contact us](mailto:cloud@zenml.io) or check out our [documentation](https://docs.zenml.io). diff --git a/docs/book/getting-started/zenml-pro/on-prem-deployment-helm.md b/docs/book/getting-started/zenml-pro/on-prem-deployment-helm.md new file mode 100644 index 00000000000..b47c88c839d --- /dev/null +++ b/docs/book/getting-started/zenml-pro/on-prem-deployment-helm.md @@ -0,0 +1,863 @@ +--- +description: Deploy ZenML Pro Air-gapped on Kubernetes with Helm - complete self-hosted setup with no external dependencies. +layout: + title: + visible: true + description: + visible: true + tableOfContents: + visible: true + outline: + visible: true + pagination: + visible: true +--- + +# Self-hosted Deployment on Kubernetes with Helm + +This guide provides step-by-step instructions for deploying ZenML Pro in a fully air-gapped setup on Kubernetes using Helm charts. In an air-gapped deployment, all components run within your infrastructure with zero external dependencies. + +## Architecture Overview + +All components run entirely within your Kubernetes cluster and infrastructure: + +``` +┌──────────────────────────────────────────────────┐ +│ Your Air-gapped Infrastructure │ +│ │ +│ ┌────────────────────────────────────────────┐ │ +│ │ Kubernetes Cluster │ │ +│ │ │ │ +│ │ ┌─────────────────────────────────────┐ │ │ +│ │ │ ZenML Pro Control Plane │ │ │ +│ │ │ - Authentication & Authorization │ │ │ +│ │ │ - RBAC Management │ │ │ +│ │ │ - Dashboard │ │ │ +│ │ └─────────────────────────────────────┘ │ │ +│ │ │ │ +│ │ ┌─────────────────────────────────────┐ │ │ +│ │ │ ZenML Workspace Servers │ │ │ +│ │ │ (one or more) │ │ │ +│ │ └─────────────────────────────────────┘ │ │ +│ │ │ │ +│ │ ┌─────────────────────────────────────┐ │ │ +│ │ │ Load Balancer / Ingress │ │ │ +│ │ │ (HTTPS with internal CA) │ │ │ +│ │ └─────────────────────────────────────┘ │ │ +│ └────────────────────────────────────────────┘ │ +│ │ +│ ┌────────────────────────────────────────────┐ │ +│ │ PostgreSQL Database │ │ +│ │ (for metadata storage) │ │ +│ └────────────────────────────────────────────┘ │ +│ │ +│ ┌────────────────────────────────────────────┐ │ +│ │ Internal Docker Registry │ │ +│ │ (for container images) │ │ +│ └────────────────────────────────────────────┘ │ +│ │ +│ ┌────────────────────────────────────────────┐ │ +│ │ Object Storage / NFS │ │ +│ │ (for artifacts & backups) │ │ +│ └────────────────────────────────────────────┘ │ +│ │ +└──────────────────────────────────────────────────┘ + 🔒 Completely Isolated - No External Access +``` + +## Prerequisites + +Before starting, you need: + +**Infrastructure:** +- Kubernetes cluster (1.24+) within your air-gapped network +- PostgreSQL database (12+) for metadata storage +- Internal Docker registry (Harbor, Quay, Artifactory, etc.) +- Load balancer or Ingress controller for HTTPS +- NFS or object storage for artifacts (optional) + +**Network:** +- Internal DNS resolution +- TLS certificates signed by your internal CA +- Network connectivity between cluster components + +**Tools (on a machine with internet access for initial setup):** +- Docker +- Helm (3.0+) +- Access to pull ZenML Pro images from private registries (credentials from ZenML) + +## Step 1: Prepare Offline Artifacts + +This step is performed on a machine with internet access, then transferred to your air-gapped environment. + +### 1.1 Pull Container Images + +On a machine with internet access and access to the ZenML Pro container registries: + +1. Authenticate to the ZenML Pro container registries (AWS ECR or GCP Artifact Registry) + - Use credentials provided by ZenML Support + - Follow registry-specific authentication procedures + +2. Pull all required images: + - **Pro Control Plane images:** + - `715803424590.dkr.ecr.eu-west-1.amazonaws.com/zenml-pro-api:
+
+## Related Resources
+
+- [System Architecture Overview](../system-architectures.md#zenml-pro-saas-architecture)
+- [Deployment Scenarios Overview](deployments-overview.md)
+- [Hybrid SaaS Deployment](hybrid-deployment.md)
+- [Self-hosted Deployment](self-hosted-deployment.md)
+- [Workload Managers](workload-managers.md)
+- [Security & Compliance](README.md#security--compliance)
+
+## Get Started
+
+Ready to get started with ZenML Pro SaaS?
+
+[Book a Demo](https://www.zenml.io/book-your-demo)
+
+Have questions? [Contact us](mailto:cloud@zenml.io) or check out our [documentation](https://docs.zenml.io).
diff --git a/docs/book/getting-started/zenml-pro/self-hosted-deployment-helm.md b/docs/book/getting-started/zenml-pro/self-hosted-deployment-helm.md
new file mode 100644
index 00000000000..7d1eaf956a8
--- /dev/null
+++ b/docs/book/getting-started/zenml-pro/self-hosted-deployment-helm.md
@@ -0,0 +1,863 @@
+---
+description: Deploy ZenML Pro Self-hosted on Kubernetes with Helm - complete self-hosted setup with no external dependencies.
+layout:
+ title:
+ visible: true
+ description:
+ visible: true
+ tableOfContents:
+ visible: true
+ outline:
+ visible: true
+ pagination:
+ visible: true
+---
+
+# Self-hosted Deployment on Kubernetes with Helm
+
+This guide provides step-by-step instructions for deploying ZenML Pro in a fully air-gapped setup on Kubernetes using Helm charts. In an air-gapped deployment, all components run within your infrastructure with zero external dependencies.
+
+## Architecture Overview
+
+All components run entirely within your Kubernetes cluster and infrastructure:
+
+```
+┌──────────────────────────────────────────────────┐
+│ Your Air-gapped Infrastructure │
+│ │
+│ ┌────────────────────────────────────────────┐ │
+│ │ Kubernetes Cluster │ │
+│ │ │ │
+│ │ ┌─────────────────────────────────────┐ │ │
+│ │ │ ZenML Pro Control Plane │ │ │
+│ │ │ - Authentication & Authorization │ │ │
+│ │ │ - RBAC Management │ │ │
+│ │ │ - Dashboard │ │ │
+│ │ └─────────────────────────────────────┘ │ │
+│ │ │ │
+│ │ ┌─────────────────────────────────────┐ │ │
+│ │ │ ZenML Workspace Servers │ │ │
+│ │ │ (one or more) │ │ │
+│ │ └─────────────────────────────────────┘ │ │
+│ │ │ │
+│ │ ┌─────────────────────────────────────┐ │ │
+│ │ │ Load Balancer / Ingress │ │ │
+│ │ │ (HTTPS with internal CA) │ │ │
+│ │ └─────────────────────────────────────┘ │ │
+│ └────────────────────────────────────────────┘ │
+│ │
+│ ┌────────────────────────────────────────────┐ │
+│ │ PostgreSQL Database │ │
+│ │ (for metadata storage) │ │
+│ └────────────────────────────────────────────┘ │
+│ │
+│ ┌────────────────────────────────────────────┐ │
+│ │ Internal Docker Registry │ │
+│ │ (for container images) │ │
+│ └────────────────────────────────────────────┘ │
+│ │
+│ ┌────────────────────────────────────────────┐ │
+│ │ Object Storage / NFS │ │
+│ │ (for artifacts & backups) │ │
+│ └────────────────────────────────────────────┘ │
+│ │
+└──────────────────────────────────────────────────┘
+ 🔒 Completely Isolated - No External Access
+```
+
+## Prerequisites
+
+Before starting, you need:
+
+**Infrastructure:**
+- Kubernetes cluster (1.24+) within your air-gapped network
+- PostgreSQL database (12+) for metadata storage
+- Internal Docker registry (Harbor, Quay, Artifactory, etc.)
+- Load balancer or Ingress controller for HTTPS
+- NFS or object storage for artifacts (optional)
+
+**Network:**
+- Internal DNS resolution
+- TLS certificates signed by your internal CA
+- Network connectivity between cluster components
+
+**Tools (on a machine with internet access for initial setup):**
+- Docker
+- Helm (3.0+)
+- Access to pull ZenML Pro images from private registries (credentials from ZenML)
+
+## Step 1: Prepare Offline Artifacts
+
+This step is performed on a machine with internet access, then transferred to your air-gapped environment.
+
+### 1.1 Pull Container Images
+
+On a machine with internet access and access to the ZenML Pro container registries:
+
+1. Authenticate to the ZenML Pro container registries (AWS ECR or GCP Artifact Registry)
+ - Use credentials provided by ZenML Support
+ - Follow registry-specific authentication procedures
+
+2. Pull all required images:
+ - **Pro Control Plane images:**
+ - `715803424590.dkr.ecr.eu-west-1.amazonaws.com/zenml-pro-api:- Authentication & Authorization
- RBAC Management
- Workspace Coordination
- Pro Metadata Store"] + + subgraph workspaces[" "] + direction LR + ws1["Workspace 1
- Server
- Metadata
- Secrets"] + ws2["Workspace 2
- Server
- Metadata
- Secrets"] + wsn["...
- Server
- Metadata
- Secrets"] + end + + compute["Your Compute & Storage Resources
- Kubernetes / VMs / Cloud
- Artifact Stores
- ML Data & Models"] + end + + control_plane --> ws1 + control_plane --> ws2 + control_plane --> wsn + ws1 --> compute + ws2 --> compute + wsn --> compute + + note[/"⚠️ No External Communication Required"/] + infra --- note +``` + +{% hint style="success" %} +**Complete data sovereignty**: Zero data leaves your environment. All components, metadata, and ML artifacts remain within your infrastructure boundaries. +{% endhint %} + +### Data Flow + +For a detailed explanation of the common pipeline execution data flow across all deployment scenarios, see [Common Pipeline Execution Data Flow](deployments-overview.md#common-pipeline-execution-data-flow) in the Deployment Scenarios Overview. + +In Self-hosted deployment, users authenticate via your internal identity provider (LDAP/AD/OIDC), and the control plane (running in your infrastructure) handles both authentication and RBAC. All communication happens entirely within your infrastructure boundary with zero external dependencies or internet connectivity required. + +## Key Benefits + +### 🔒 Maximum Security & Control + +- **Complete air-gap**: No internet connectivity required for operation +- **Zero external dependencies**: All components self-contained +- **Custom security policies**: Full control over all security configurations +- **Network isolation**: Operates within your security perimeter +- **Audit compliance**: Complete logging and monitoring within your infrastructure + +### 🏛️ Regulatory Compliance + +- **Data residency**: All data stays within your jurisdiction +- **ITAR/EAR compliance**: Suitable for controlled data environments +- **HIPAA/GDPR ready**: Meet healthcare and privacy regulations +- **Government/Defense**: Suitable for classified environments +- **Financial services**: Meet banking and financial regulations + +### 🎯 Enterprise Control + +- **Custom identity provider**: Integrate with your LDAP/AD/OIDC +- **Infrastructure flexibility**: Deploy on any infrastructure (cloud, on-prem, edge) +- **Version control**: Control update schedules and versions +- **Backup strategy**: Implement your own backup and DR policies +- **Resource optimization**: Full control over resource allocation and costs + +### 🛡️ Certified & Documented + +- **SOC 2 & ISO 27001 certified software**: Meets enterprise security and compliance benchmarks for your peace of mind +- **Vulnerability Assessment Reports**: Available on request +- **Software Bill of Materials (SBOM)**: Complete dependency inventory +- **Architecture documentation**: Comprehensive deployment guides + +## Ideal Use Cases + +Self-hosted deployment is essential for: + +- **Government and defense** organizations with classified data requirements +- **Regulated industries** (healthcare, finance) with strict data residency requirements +- **Organizations in restricted regions** with limited or no internet connectivity +- **Research institutions** handling sensitive or proprietary research data +- **Critical infrastructure** operators requiring isolated systems +- **Companies with ITAR/EAR compliance** requirements +- **Enterprises with zero-trust policies** prohibiting external communication +- **Organizations requiring full control** over all aspects of their MLOps platform + +## Deployment Options + +### On-Premises Data Center + +Deploy on your own hardware: +- Physical servers or private cloud +- Complete infrastructure control +- Integration with existing systems +- Custom hardware configurations + +### Private Cloud (AWS, Azure, GCP) + +Deploy in isolated cloud VPC: +- No internet gateway +- Private networking only +- Use cloud-native services +- Leverage cloud scalability within your boundary + +### Hybrid Multi-Cloud + +Deploy across multiple environments: +- On-premises + private cloud +- Multi-region for DR +- Edge + datacenter hybrid +- Maintain complete isolation across all environments + +### Edge Deployments + +Deploy at edge locations: +- Manufacturing facilities +- Remote research stations +- Mobile/tactical deployments +- Disconnected field operations + +## Deployment Architecture + +### Architecture Diagram + + + +The diagram above illustrates a complete Self-hosted ZenML Pro deployment with all components running within your organization's VPC. This architecture ensures zero external communication while providing full enterprise MLOps capabilities. + +### Architecture Components + +**Client SDK** (top center): +The ZenML Python SDK runs on developer laptops, CI/CD systems, or notebooks. It communicates with all layers to: +- Authenticate users via your Identity Provider +- Submit pipeline runs to workspaces +- Push Docker images to your Container Registry +- Access the Organization Platform Layer components + +**Organization Platform Layer** (left, pink): +Your existing ML infrastructure components that ZenML integrates with: +- **Container Registry**: Store pipeline Docker images (AWS ECR, Dockerhub, Google Artifact Registry, Azure Container Registry) +- **Artifact Store**: Store ML artifacts, models, and datasets (S3, GCS, Azure Blob Storage, ADLS) +- **Code Repository**: Version control for pipeline code (GitHub Enterprise, GitLab, Bitbucket) +- **Orchestrator**: Execute pipeline workloads (Vertex AI, Sagemaker, AzureML, Kubernetes) + +**Infrastructure Layer** (top, cyan): +Core infrastructure services: +- **Identity Provider**: LDAP, Active Directory, or OIDC provider for user authentication +- **Load Balancer**: Distributes traffic to ZenML services for high availability + +**ZenML Control Plane** (center, blue): +The management layer running in Kubernetes: +- **ZenML FE**: React-based Pro dashboard for pipeline visualization and model management +- **ZenML Control Plane**: Coordinates workspaces, handles authentication/RBAC, manages organization settings + +**ZenML Application Plane** (center, purple): +Individual workspace servers running in Kubernetes: +- **Multiple Workspaces**: Isolated environments for different teams (DS Team 1, DS Team 2, etc.) +- Each workspace has its own server instance, metadata database, and secrets store +- Workspaces are orchestrated by the Control Plane but run independently + +**ZenML Storage Plane** (bottom, pink): +Persistent storage for ZenML services: +- **Secret Store**: Vault or cloud secrets manager for storing credentials securely +- **Database**: PostgreSQL or MySQL for storing workspace metadata, pipeline runs, and control plane data + +### Data Flow + +All arrows in the diagram represent communication flows that occur entirely within your VPC: +1. Client SDK authenticates with Identity Provider +2. Client SDK connects to ZenML Control Plane for workspace access +3. Control Plane manages and coordinates workspaces +4. Workspaces orchestrate pipeline execution on your Orchestrator +5. Pipelines write artifacts to your Artifact Store +6. Workspaces store metadata in the Database +7. All components access secrets from the Secret Store + +**Key Security Feature**: The entire system operates without any external internet connectivity. All Docker images, dependencies, and updates are transferred to your environment through secure offline channels. + +### High Availability Configuration + +For mission-critical deployments: +- **Active-active** control plane for zero downtime +- **Database replication** for metadata stores +- **Load balancers** for workspace servers +- **Backup sites** for disaster recovery +- **Monitoring and alerting** for all components + +## Pre-requisites + +Before deployment, ensure you have: + +#### Infrastructure Requirements +- Kubernetes cluster (recommended) or VM infrastructure +- PostgreSQL database(s) for metadata storage +- Object storage or NFS for artifacts +- Load balancer for HA configurations +- Identity provider (LDAP/AD/OIDC) + +#### Network Requirements +- Internal DNS resolution +- SSL/TLS certificates (internal CA) +- Network connectivity between components +- Firewall rules for inter-component communication + +#### Resource Requirements +```yaml +# Minimum requirements +Control Plane: + CPU: 4 cores + Memory: 16GB RAM + Storage: 100GB + +Per Workspace: + CPU: 2 cores + Memory: 8GB RAM + Storage: 50GB + metadata + +Database: + CPU: 4 cores + Memory: 16GB RAM + Storage: 500GB (scalable) +``` + +## Operations & Maintenance + +### Updates & Upgrades + +ZenML provides new versions as offline bundles: + +1. **Receive new bundle**: Typically by pulling our Docker images via your approved transfer method +2. **Review release notes and compatibility notes**: Carefully review the release notes and any migration instructions included in the offline bundle to understand all changes, requirements, and potential impacts. Assess required infrastructure or configuration updates and note any changes in CI/CD actions or deployment processes before proceeding. +3. **Test in staging**: Deploy to test environment first +4. **Backup current state**: Database and configuration backups +5. **Apply updates**: Using Helm upgrade commands, or update your deployment using Terraform or other Infrastructure-as-Code (IaC) tools. +6. **Verify functionality**: Run health checks and tests +7. **Monitor**: Watch for any issues post-upgrade + + +### Disaster Recovery + +Plan for disaster scenarios: + +1. **Database replication**: PostgreSQL streaming replication to backup site +2. **Artifact replication**: Sync artifact stores to DR location +3. **Configuration backup**: Version-controlled infrastructure as code +4. **Runbook**: Document DR procedures +5. **Regular testing**: Test DR procedures quarterly + +## Security Hardening + +### Network Security + +- **Network segmentation**: Isolate ZenML components in dedicated network segments +- **Firewall rules**: Restrict traffic to only required ports +- **TLS everywhere**: Encrypt all communication +- **Certificate management**: Use internal CA for certificate issuance + + +### Access Control + +- **Principle of least privilege**: Grant minimal required permissions +- **Service accounts**: Use dedicated service accounts for automation +- **Audit logging**: Log all authentication and authorization events + +### Container Security + +- **Image scanning**: Scan all container images before deployment +- **Runtime security**: Monitor container behavior +- **Pod security policies**: Enforce security standards +- **Resource limits**: Prevent resource exhaustion attacks + +## Support & Documentation + +### What ZenML Provides + +- **Deployment packages**: Complete offline installation bundles +- **Documentation**: Comprehensive setup and operation guides +- **SBOM**: Full software bill of materials for compliance +- **Vulnerability reports**: Security assessment documentation +- **Architecture consultation**: Pre-deployment planning support +- **Deployment assistance**: Guidance during initial setup +- **Update packages**: New versions as offline bundles + +### What You Manage + +- **Infrastructure**: Hardware, networking, storage +- **Day-to-day operations**: Monitoring, backups, user management +- **Security policies**: Firewall rules, access controls +- **Compliance**: Audit logs, security assessments +- **Updates**: Applying new versions using provided bundles + +### Support Model + +Contact [cloud@zenml.io](mailto:cloud@zenml.io) for: +- Pre-sales architecture consultation +- Deployment planning and sizing +- Security documentation requests +- Offline support packages +- Update and upgrade assistance + +## Licensing + +Air-gapped deployments are provided under commercial software license agreements, with license fees and terms defined on a per-customer basis. Each contract includes detailed license terms and conditions appropriate to the deployment. + +## Security Documentation + +Available on request for compliance and security reviews: + +- ✅ **Vulnerability Assessment Reports**: Full security analysis +- ✅ **Software Bill of Materials (SBOM)**: Complete dependency list +- ✅ **Architecture security review**: Threat model and mitigations +- ✅ **Compliance mappings**: NIST, CIS, GDPR, HIPAA guidance +- ✅ **Security hardening guide**: Best practices for your deployment + +## Comparison with Other Deployments + +| Feature | SaaS | Hybrid SaaS | Self-hosted | +|---------|------|-------------|------------| +| Internet Required | Yes (metadata) | Yes (control plane) | **No** | +| Setup Time | Minutes | Hours/Days | Days/Weeks | +| Maintenance | Zero | Partial | **Full control** | +| Data Location | Mixed | Your infra | **100% yours** | +| User Management | ZenML | ZenML | **Your IDP** | +| Update Control | Automatic | Automatic CP | **You decide** | +| Customization | Limited | Moderate | **Complete** | +| Best For | Fast start | Balance | **Max security** | + +[Compare all deployment options →](README.md#deployment-scenarios) + +## Migration Path + +### From ZenML OSS to Self-hosted Pro + +If you're interested in migrating from ZenML OSS to a Self-hosted Pro deployment, we're here to help guide you through every step of the process. Migration paths are highly dependent on your specific environment, infrastructure setup, and current ZenML OSS deployment configuration. + +It's possible to migrate existing stacks or even existing metadata from existing OSS deployments. We can figure out how and what to migrate together in a call. + +**Next steps:** + +- [Book a migration consultation →](https://www.zenml.io/book-your-demo) +- Or email us at [cloud@zenml.io](mailto:cloud@zenml.io) + +Your ZenML representative will work with you to assess your current setup, understand your Self-hosted requirements, and provide a tailored migration plan that fits your environment. + +### From Other Pro Deployments + +If you're moving from SaaS or Hybrid to Self-hosted, migration paths can vary significantly depending on your organization's size, data residency requirements, and current ZenML setup. We recommend discussing your plans with a ZenML solutions architect. + +**Next steps:** + +- [Book a migration consultation →](https://www.zenml.io/book-your-demo) +- Or email us at [cloud@zenml.io](mailto:cloud@zenml.io) + +Your ZenML representative will provide you with a tailored migration checklist, technical documentation, and direct support to ensure a smooth transition with minimal downtime. + +## Detailed Architecture Diagram + +
.png)
+
+
+## Related Resources
+
+- [System Architecture Overview](../system-architectures.md#zenml-pro-self-hosted-architecture)
+- [Deployment Scenarios Overview](deployments-overview.md)
+- [SaaS Deployment](saas-deployment.md)
+- [Hybrid SaaS Deployment](hybrid-deployment.md)
+- [Workload Managers](workload-managers.md)
+- [Self-hosted Deployment Guide](self-hosted.md)
+- [Security & Compliance](README.md#security--compliance)
+
+## Get Started
+
+Ready to deploy ZenML Pro in a Self-hosted environment?
+
+[Book a Demo](https://www.zenml.io/book-your-demo){ .md-button .md-button--primary }
+
+Have questions? [Contact us](mailto:cloud@zenml.io) for detailed deployment planning.
diff --git a/docs/book/getting-started/zenml-pro/toc.md b/docs/book/getting-started/zenml-pro/toc.md
index 79f4ff07e21..f0fc262337f 100644
--- a/docs/book/getting-started/zenml-pro/toc.md
+++ b/docs/book/getting-started/zenml-pro/toc.md
@@ -4,7 +4,13 @@
## Deployments
-* [Self-hosted deployment](self-hosted.md)
+* [Deployment Scenarios Overview](deployments-overview.md)
+* [SaaS Deployment](saas-deployment.md)
+* [Hybrid SaaS Deployment](hybrid-deployment.md)
+ * [Kubernetes with Helm](hybrid-deployment-helm.md)
+ * [AWS ECS](hybrid-deployment-ecs.md)
+* [Self-hosted Deployment](self-hosted-deployment.md)
+ * [Kubernetes with Helm](self-hosted-deployment-helm.md)
## Core Concepts
@@ -13,6 +19,7 @@
* [Workspaces](workspaces.md)
* [Projects](projects.md)
* [Teams](teams.md)
+* [Workload Managers](workload-managers.md)
## Access Management
diff --git a/docs/book/getting-started/zenml-pro/workload-managers.md b/docs/book/getting-started/zenml-pro/workload-managers.md
new file mode 100644
index 00000000000..2c1857b75b3
--- /dev/null
+++ b/docs/book/getting-started/zenml-pro/workload-managers.md
@@ -0,0 +1,391 @@
+---
+description: Understand workload managers and how they enable running pipelines from the ZenML Pro UI.
+icon: microchip
+---
+
+# Workload Managers
+
+Workload managers are built into the ZenML Pro workspace container. They enable you to run pipeline snapshots directly from the ZenML Pro UI by allowing the workspace to orchestrate pipeline execution on your infrastructure. Without a workload manager configured, your workspace can only be used for monitoring and analyzing completed pipeline runs. With one configured, you gain the ability to trigger and execute pipelines interactively.
+
+{% hint style="info" %}
+This feature is available in [all ZenML Pro deployment scenarios](deployments-overview.md) (SaaS, Hybrid, and Self-hosted).
+{% endhint %}
+
+## Architecture
+
+The ZenML Pro workspace container includes workload manager implementations. You configure which implementation to use through environment variables passed to the workspace. The workspace then uses that implementation to coordinate pipeline execution with your infrastructure.
+
+### Execution Flow
+
+1. **User triggers a snapshot from the ZenML Pro UI**: You select a pipeline snapshot and click "Run"
+2. **ZenML workspace receives the request**: Your ZenML Pro workspace (running in your workspace, whether SaaS, Hybrid, or Self-hosted) receives the execution request.
+3. **Workload manager implementation handles orchestration**: The configured workload manager implementation (Kubernetes, AWS, or GCP) translates the request into infrastructure-specific commands.
+4. **Runner pod/task is created**: The workload manager creates a Kubernetes pod, ECS task, or equivalent compute unit on your infrastructure.
+5. **Pipeline executes**: The runner pulls the pipeline code, executes the steps, and streams logs back to the workspace.
+6. **Results are captured**: Artifacts, metrics, and run metadata are stored in your configured artifact store and metadata backend.
+
+## How Workload Managers Are Configured
+
+Workload managers are enabled by setting environment variables on the ZenML Pro workspace container. Each implementation requires a specific set of environment variables that tell the workspace:
+
+- Which workload manager implementation to use
+- Where to create runner pods/tasks (namespace, cluster, region)
+- How to access container registries and storage
+- What permissions and resources runners should have
+
+All configuration happens within a single workspace deployment—no separate services are needed.
+
+## Supported Implementations
+
+ZenML Pro supports three workload manager implementations:
+
+### 1. Kubernetes Workload Manager
+
+The simplest implementation, suitable for any Kubernetes cluster (EKS, GKE, AKS, self-managed).
+
+**Environment Variables:**
+
+```yaml
+ZENML_SERVER_WORKLOAD_MANAGER_IMPLEMENTATION_SOURCE: zenml_cloud_plugins.kubernetes_workload_manager.KubernetesWorkloadManager
+ZENML_KUBERNETES_WORKLOAD_MANAGER_NAMESPACE: zenml-workload-manager
+ZENML_KUBERNETES_WORKLOAD_MANAGER_SERVICE_ACCOUNT: zenml-runner
+ZENML_KUBERNETES_WORKLOAD_MANAGER_RUNNER_IMAGE: 715803424590.dkr.ecr.eu-central-1.amazonaws.com/zenml-pro-server:0.73.0
+ZENML_KUBERNETES_WORKLOAD_MANAGER_POD_RESOURCES: '{"requests": {"cpu": "500m", "memory": "512Mi"}, "limits": {"cpu": "2000m", "memory": "2Gi"}}'
+ZENML_KUBERNETES_WORKLOAD_MANAGER_TTL_SECONDS_AFTER_FINISHED: 86400
+ZENML_SERVER_MAX_CONCURRENT_TEMPLATE_RUNS: 5
+```
+
+**Requirements:**
+- Kubernetes cluster (1.24+)
+- Service account with permissions to create/manage pods in a dedicated namespace
+- Network connectivity from cluster to your ZenML workspace
+- Access to a container registry with ZenML runner images
+
+**How it works:**
+- The workspace uses the Kubernetes API to create pods in the specified namespace
+- Pods run under the specified service account, inheriting cluster network access
+- Completed pods are automatically cleaned up after the TTL expires
+
+**Use cases:**
+- Self-managed ZenML workspaces on Kubernetes (Hybrid or Self-hosted)
+- Teams already running Kubernetes infrastructure
+- Minimal setup complexity
+
+### 2. AWS Kubernetes Workload Manager
+
+A specialized implementation for EKS that integrates with AWS services (ECR for images, S3 for logs, IAM for permissions).
+
+**Environment Variables:**
+
+```yaml
+ZENML_SERVER_WORKLOAD_MANAGER_IMPLEMENTATION_SOURCE: zenml_cloud_plugins.aws_kubernetes_workload_manager.AWSKubernetesWorkloadManager
+ZENML_KUBERNETES_WORKLOAD_MANAGER_NAMESPACE: zenml-workload-manager
+ZENML_KUBERNETES_WORKLOAD_MANAGER_SERVICE_ACCOUNT: zenml-runner
+ZENML_KUBERNETES_WORKLOAD_MANAGER_BUILD_RUNNER_IMAGE: "true"
+ZENML_KUBERNETES_WORKLOAD_MANAGER_DOCKER_REGISTRY: 