From b195cbbf30602bd00f203df7e632ffd1c42ed5d9 Mon Sep 17 00:00:00 2001 From: Safoine El khabich Date: Wed, 3 Dec 2025 11:19:26 +0100 Subject: [PATCH 1/5] init --- .../component-guide/artifact-stores/README.md | 2 + .../artifact-stores/alibaba-oss.md | 92 +++++++++++++++++++ .../component-guide/artifact-stores/minio.md | 89 ++++++++++++++++++ docs/book/component-guide/toc.md | 2 + 4 files changed, 185 insertions(+) create mode 100644 docs/book/component-guide/artifact-stores/alibaba-oss.md create mode 100644 docs/book/component-guide/artifact-stores/minio.md diff --git a/docs/book/component-guide/artifact-stores/README.md b/docs/book/component-guide/artifact-stores/README.md index 0bc26f7fb3c..c2d6ec82c90 100644 --- a/docs/book/component-guide/artifact-stores/README.md +++ b/docs/book/component-guide/artifact-stores/README.md @@ -36,6 +36,8 @@ Out of the box, ZenML comes with a `local` artifact store already part of the de | [Amazon S3](s3.md) | `s3` | `s3` | `s3://` | Uses AWS S3 as an object store backend | | [Google Cloud Storage](gcp.md) | `gcp` | `gcp` | `gs://` | Uses Google Cloud Storage as an object store backend | | [Azure](azure.md) | `azure` | `azure` | `abfs://`, `az://` | Uses Azure Blob Storage as an object store backend | +| [Alibaba Cloud OSS](alibaba-oss.md) | `s3` | `s3` | `s3://` | Uses S3 integration to connect to Alibaba Cloud OSS | +| [MinIO](minio.md) | `s3` | `s3` | `s3://` | Uses S3 integration to connect to self-hosted MinIO | | [Custom Implementation](custom.md) | _custom_ | | _custom_ | Extend the Artifact Store abstraction and provide your own implementation | If you would like to see the available flavors of Artifact Stores, you can use the command: diff --git a/docs/book/component-guide/artifact-stores/alibaba-oss.md b/docs/book/component-guide/artifact-stores/alibaba-oss.md new file mode 100644 index 00000000000..7cc9fbaeb85 --- /dev/null +++ b/docs/book/component-guide/artifact-stores/alibaba-oss.md @@ -0,0 +1,92 @@ +--- +description: Storing artifacts in Alibaba Cloud Object Storage Service (OSS). +--- + +# Alibaba Cloud OSS + +Alibaba Cloud Object Storage Service (OSS) is an S3-compatible object storage service. Since OSS provides an S3-compatible API, you can use ZenML's S3 Artifact Store integration to connect to Alibaba Cloud OSS. + +### When would you want to use it? + +You should use the Alibaba Cloud OSS Artifact Store when you want to store your ZenML artifacts in Alibaba Cloud infrastructure, particularly if: + +* You're already using Alibaba Cloud services +* You need to store artifacts in specific geographic regions served by Alibaba Cloud +* You want to leverage Alibaba Cloud's pricing and performance characteristics + +### How do you deploy it? + +Since Alibaba Cloud OSS is S3-compatible, you'll use the S3 integration. First, install it: + +```shell +zenml integration install s3 -y +``` + +You'll also need to create an OSS bucket and obtain your access credentials from the Alibaba Cloud console. + +### How do you configure it? + +To use Alibaba Cloud OSS with ZenML, you need to configure the S3 Artifact Store with specific settings for OSS compatibility: + +{% tabs %} +{% tab title="Using a ZenML Secret (recommended)" %} + +First, create a ZenML secret with your Alibaba Cloud credentials: + +```shell +zenml secret create alibaba_secret \ + --aws_access_key_id='' \ + --aws_secret_access_key='' +``` + +Then register the artifact store with the required OSS configuration: + +```shell +zenml artifact-store register alibaba_store -f s3 \ + --path='s3://your-bucket-name' \ + --authentication_secret=alibaba_secret \ + --client_kwargs='{"endpoint_url": "https://oss-.aliyuncs.com"}' \ + --config_kwargs='{"signature_version": "s3", "s3": {"addressing_style": "virtual"}}' +``` +{% endtab %} + +{% tab title="Inline credentials" %} + +You can also provide credentials directly (not recommended for production): + +```shell +zenml artifact-store register alibaba_store -f s3 \ + --path='s3://your-bucket-name' \ + --aws_access_key_id='' \ + --aws_secret_access_key='' \ + --client_kwargs='{"endpoint_url": "https://oss-.aliyuncs.com"}' \ + --config_kwargs='{"signature_version": "s3", "s3": {"addressing_style": "virtual"}}' +``` +{% endtab %} +{% endtabs %} + +{% hint style="warning" %} +**Important:** When using Alibaba Cloud OSS, you must set the following `config_kwargs`: + +```json +{"signature_version": "s3", "s3": {"addressing_style": "virtual"}} +``` + +This is required for proper compatibility with Alibaba Cloud OSS's S3 API implementation. +{% endhint %} + +Replace `` with your OSS region (e.g., `eu-central-1`, `cn-hangzhou`, `ap-southeast-1`). You can find the list of available regions and their endpoints in the [Alibaba Cloud OSS documentation](https://www.alibabacloud.com/help/en/oss/user-guide/regions-and-endpoints). + +Finally, add the artifact store to your stack: + +```shell +zenml stack register custom_stack -a alibaba_store ... --set +``` + +### How do you use it? + +Using the Alibaba Cloud OSS Artifact Store is no different from [using any other flavor of Artifact Store](./#how-to-use-it). ZenML handles the S3-compatible API translation automatically. + +For more details on the S3 Artifact Store configuration options, refer to the [S3 Artifact Store documentation](s3.md). + +
ZenML Scarf
diff --git a/docs/book/component-guide/artifact-stores/minio.md b/docs/book/component-guide/artifact-stores/minio.md new file mode 100644 index 00000000000..d4fa7a9e55a --- /dev/null +++ b/docs/book/component-guide/artifact-stores/minio.md @@ -0,0 +1,89 @@ +--- +description: Storing artifacts in MinIO object storage. +--- + +# MinIO + +[MinIO](https://min.io/) is a high-performance, S3-compatible object storage system. Since MinIO provides a fully S3-compatible API, you can use ZenML's S3 Artifact Store integration to connect to MinIO. + +### When would you want to use it? + +You should use the MinIO Artifact Store when: + +* You want to run a self-hosted object storage solution +* You need S3-compatible storage in your own data center or private cloud +* You're running ZenML in a Kubernetes environment and want co-located storage +* You want to avoid cloud vendor lock-in while maintaining S3 API compatibility + +### How do you deploy it? + +Since MinIO is S3-compatible, you'll use the S3 integration. First, install it: + +```shell +zenml integration install s3 -y +``` + +You'll also need a running MinIO instance. MinIO can be deployed in various ways: + +* **Docker**: `docker run -p 9000:9000 -p 9001:9001 minio/minio server /data --console-address ":9001"` +* **Kubernetes**: Using the [MinIO Operator](https://min.io/docs/minio/kubernetes/upstream/) +* **Binary**: Download from [MinIO's website](https://min.io/download) + +### How do you configure it? + +To use MinIO with ZenML, configure the S3 Artifact Store with your MinIO endpoint: + +{% tabs %} +{% tab title="Using a ZenML Secret (recommended)" %} + +First, create a ZenML secret with your MinIO credentials: + +```shell +zenml secret create minio_secret \ + --aws_access_key_id='' \ + --aws_secret_access_key='' +``` + +Then register the artifact store: + +```shell +zenml artifact-store register minio_store -f s3 \ + --path='s3://your-bucket-name' \ + --authentication_secret=minio_secret \ + --client_kwargs='{"endpoint_url": "http://minio.example.com:9000"}' +``` +{% endtab %} + +{% tab title="Inline credentials" %} + +You can also provide credentials directly (not recommended for production): + +```shell +zenml artifact-store register minio_store -f s3 \ + --path='s3://your-bucket-name' \ + --aws_access_key_id='' \ + --aws_secret_access_key='' \ + --client_kwargs='{"endpoint_url": "http://minio.example.com:9000"}' +``` +{% endtab %} +{% endtabs %} + +Replace `http://minio.example.com:9000` with your actual MinIO endpoint. If you're running MinIO locally for development, this might be `http://localhost:9000`. + +{% hint style="info" %} +If your MinIO instance uses HTTPS with a self-signed certificate, you may need to configure SSL verification. Consult the [S3 Artifact Store documentation](s3.md#advanced-configuration) for advanced configuration options. +{% endhint %} + +Finally, add the artifact store to your stack: + +```shell +zenml stack register custom_stack -a minio_store ... --set +``` + +### How do you use it? + +Using the MinIO Artifact Store is no different from [using any other flavor of Artifact Store](./#how-to-use-it). ZenML handles the S3-compatible API translation automatically. + +For more details on the S3 Artifact Store configuration options, refer to the [S3 Artifact Store documentation](s3.md). + +
ZenML Scarf
diff --git a/docs/book/component-guide/toc.md b/docs/book/component-guide/toc.md index be1e3f1f76e..48cd2569259 100644 --- a/docs/book/component-guide/toc.md +++ b/docs/book/component-guide/toc.md @@ -32,6 +32,8 @@ * [Amazon Simple Cloud Storage (S3)](artifact-stores/s3.md) * [Google Cloud Storage (GCS)](artifact-stores/gcp.md) * [Azure Blob Storage](artifact-stores/azure.md) + * [Alibaba Cloud OSS](artifact-stores/alibaba-oss.md) + * [MinIO](artifact-stores/minio.md) * [Develop a custom artifact store](artifact-stores/custom.md) * [Container Registries](container-registries/README.md) * [Default Container Registry](container-registries/default.md) From e3fe974083c5d2ace8672e0b6928bcc565b248e8 Mon Sep 17 00:00:00 2001 From: Safoine El khabich Date: Wed, 3 Dec 2025 12:00:44 +0100 Subject: [PATCH 2/5] update docs --- .../artifact-stores/alibaba-oss.md | 36 ++++++------------- .../component-guide/artifact-stores/minio.md | 13 ------- 2 files changed, 11 insertions(+), 38 deletions(-) diff --git a/docs/book/component-guide/artifact-stores/alibaba-oss.md b/docs/book/component-guide/artifact-stores/alibaba-oss.md index 7cc9fbaeb85..500a00ae937 100644 --- a/docs/book/component-guide/artifact-stores/alibaba-oss.md +++ b/docs/book/component-guide/artifact-stores/alibaba-oss.md @@ -4,7 +4,17 @@ description: Storing artifacts in Alibaba Cloud Object Storage Service (OSS). # Alibaba Cloud OSS -Alibaba Cloud Object Storage Service (OSS) is an S3-compatible object storage service. Since OSS provides an S3-compatible API, you can use ZenML's S3 Artifact Store integration to connect to Alibaba Cloud OSS. +[Alibaba Cloud Object Storage Service (OSS)](https://www.alibabacloud.com/product/object-storage-service) is an S3-compatible object storage service. Since OSS provides an S3-compatible API, you can use ZenML's S3 Artifact Store integration to connect to [Alibaba Cloud](https://www.alibabacloud.com) OSS. + +{% hint style="warning" %} +**Important:** When using Alibaba Cloud OSS, you must set the following `config_kwargs`: + +```json +{"signature_version": "s3", "s3": {"addressing_style": "virtual"}} +``` + +This is required for proper compatibility with Alibaba Cloud OSS's S3 API implementation. +{% endhint %} ### When would you want to use it? @@ -49,32 +59,8 @@ zenml artifact-store register alibaba_store -f s3 \ --config_kwargs='{"signature_version": "s3", "s3": {"addressing_style": "virtual"}}' ``` {% endtab %} - -{% tab title="Inline credentials" %} - -You can also provide credentials directly (not recommended for production): - -```shell -zenml artifact-store register alibaba_store -f s3 \ - --path='s3://your-bucket-name' \ - --aws_access_key_id='' \ - --aws_secret_access_key='' \ - --client_kwargs='{"endpoint_url": "https://oss-.aliyuncs.com"}' \ - --config_kwargs='{"signature_version": "s3", "s3": {"addressing_style": "virtual"}}' -``` -{% endtab %} {% endtabs %} -{% hint style="warning" %} -**Important:** When using Alibaba Cloud OSS, you must set the following `config_kwargs`: - -```json -{"signature_version": "s3", "s3": {"addressing_style": "virtual"}} -``` - -This is required for proper compatibility with Alibaba Cloud OSS's S3 API implementation. -{% endhint %} - Replace `` with your OSS region (e.g., `eu-central-1`, `cn-hangzhou`, `ap-southeast-1`). You can find the list of available regions and their endpoints in the [Alibaba Cloud OSS documentation](https://www.alibabacloud.com/help/en/oss/user-guide/regions-and-endpoints). Finally, add the artifact store to your stack: diff --git a/docs/book/component-guide/artifact-stores/minio.md b/docs/book/component-guide/artifact-stores/minio.md index d4fa7a9e55a..201f7f7cf3a 100644 --- a/docs/book/component-guide/artifact-stores/minio.md +++ b/docs/book/component-guide/artifact-stores/minio.md @@ -53,19 +53,6 @@ zenml artifact-store register minio_store -f s3 \ --client_kwargs='{"endpoint_url": "http://minio.example.com:9000"}' ``` {% endtab %} - -{% tab title="Inline credentials" %} - -You can also provide credentials directly (not recommended for production): - -```shell -zenml artifact-store register minio_store -f s3 \ - --path='s3://your-bucket-name' \ - --aws_access_key_id='' \ - --aws_secret_access_key='' \ - --client_kwargs='{"endpoint_url": "http://minio.example.com:9000"}' -``` -{% endtab %} {% endtabs %} Replace `http://minio.example.com:9000` with your actual MinIO endpoint. If you're running MinIO locally for development, this might be `http://localhost:9000`. From e871194768d8eebefa3e19b7a41322106f4834ce Mon Sep 17 00:00:00 2001 From: Safoine El khabich Date: Fri, 5 Dec 2025 14:01:14 +0100 Subject: [PATCH 3/5] update secret namiong --- .../artifact-stores/alibaba-oss.md | 4 +-- .../component-guide/artifact-stores/minio.md | 4 +-- .../component-guide/artifact-stores/s3.md | 4 +-- src/zenml/secret/schemas/aws_secret_schema.py | 28 ++++++++++++++++--- 4 files changed, 30 insertions(+), 10 deletions(-) diff --git a/docs/book/component-guide/artifact-stores/alibaba-oss.md b/docs/book/component-guide/artifact-stores/alibaba-oss.md index 500a00ae937..906b58d68ea 100644 --- a/docs/book/component-guide/artifact-stores/alibaba-oss.md +++ b/docs/book/component-guide/artifact-stores/alibaba-oss.md @@ -45,8 +45,8 @@ First, create a ZenML secret with your Alibaba Cloud credentials: ```shell zenml secret create alibaba_secret \ - --aws_access_key_id='' \ - --aws_secret_access_key='' + --access_key_id='' \ + --secret_access_key='' ``` Then register the artifact store with the required OSS configuration: diff --git a/docs/book/component-guide/artifact-stores/minio.md b/docs/book/component-guide/artifact-stores/minio.md index 201f7f7cf3a..2468211cd59 100644 --- a/docs/book/component-guide/artifact-stores/minio.md +++ b/docs/book/component-guide/artifact-stores/minio.md @@ -40,8 +40,8 @@ First, create a ZenML secret with your MinIO credentials: ```shell zenml secret create minio_secret \ - --aws_access_key_id='' \ - --aws_secret_access_key='' + --access_key_id='' \ + --secret_access_key='' ``` Then register the artifact store: diff --git a/docs/book/component-guide/artifact-stores/s3.md b/docs/book/component-guide/artifact-stores/s3.md index 0726086cfef..67fda3b7a1b 100644 --- a/docs/book/component-guide/artifact-stores/s3.md +++ b/docs/book/component-guide/artifact-stores/s3.md @@ -174,8 +174,8 @@ After having set up the IAM user and generated the access key, as described in t ```shell # Store the AWS access key in a ZenML secret zenml secret create s3_secret \ - --aws_access_key_id='' \ - --aws_secret_access_key='' + --access_key_id='' \ + --secret_access_key='' # Register the S3 artifact-store and reference the ZenML secret zenml artifact-store register s3_store -f s3 \ diff --git a/src/zenml/secret/schemas/aws_secret_schema.py b/src/zenml/secret/schemas/aws_secret_schema.py index 7df719ee634..b202830863a 100644 --- a/src/zenml/secret/schemas/aws_secret_schema.py +++ b/src/zenml/secret/schemas/aws_secret_schema.py @@ -15,12 +15,32 @@ from typing import Optional +from pydantic import AliasChoices, Field + from zenml.secret.base_secret import BaseSecretSchema class AWSSecretSchema(BaseSecretSchema): - """AWS Authentication Secret Schema definition.""" + """AWS Authentication Secret Schema definition. + + This schema supports both AWS-prefixed field names (for AWS S3) and + generic field names (for S3-compatible storage like MinIO, Alibaba OSS). + + Supported field names: + - aws_access_key_id / access_key_id + - aws_secret_access_key / secret_access_key + - aws_session_token / session_token + """ - aws_access_key_id: str - aws_secret_access_key: str - aws_session_token: Optional[str] = None + aws_access_key_id: str = Field( + validation_alias=AliasChoices("aws_access_key_id", "access_key_id"), + ) + aws_secret_access_key: str = Field( + validation_alias=AliasChoices( + "aws_secret_access_key", "secret_access_key" + ), + ) + aws_session_token: Optional[str] = Field( + default=None, + validation_alias=AliasChoices("aws_session_token", "session_token"), + ) From 0622f34210487d4bec0ecddaa9676ba36a92e291 Mon Sep 17 00:00:00 2001 From: Safoine El khabich Date: Mon, 8 Dec 2025 15:07:06 +0100 Subject: [PATCH 4/5] fixes by claude --- .../component-guide/artifact-stores/alibaba-oss.md | 13 +++++++++---- docs/book/component-guide/artifact-stores/minio.md | 9 +++++---- 2 files changed, 14 insertions(+), 8 deletions(-) diff --git a/docs/book/component-guide/artifact-stores/alibaba-oss.md b/docs/book/component-guide/artifact-stores/alibaba-oss.md index 906b58d68ea..f225c3c98c7 100644 --- a/docs/book/component-guide/artifact-stores/alibaba-oss.md +++ b/docs/book/component-guide/artifact-stores/alibaba-oss.md @@ -18,11 +18,12 @@ This is required for proper compatibility with Alibaba Cloud OSS's S3 API implem ### When would you want to use it? -You should use the Alibaba Cloud OSS Artifact Store when you want to store your ZenML artifacts in Alibaba Cloud infrastructure, particularly if: +You should use the Alibaba Cloud OSS Artifact Store when: -* You're already using Alibaba Cloud services -* You need to store artifacts in specific geographic regions served by Alibaba Cloud -* You want to leverage Alibaba Cloud's pricing and performance characteristics +* Your infrastructure is already deployed on Alibaba Cloud and you want to maintain data locality +* You require artifact storage in specific geographic regions served by Alibaba Cloud (China, Asia-Pacific, Europe, Middle East) +* You need S3-compatible object storage with Alibaba Cloud's pricing model and service level agreements +* Compliance requirements mandate data residency in Alibaba Cloud regions ### How do you deploy it? @@ -38,6 +39,10 @@ You'll also need to create an OSS bucket and obtain your access credentials from To use Alibaba Cloud OSS with ZenML, you need to configure the S3 Artifact Store with specific settings for OSS compatibility: +{% hint style="info" %} +Alibaba Cloud OSS does not support ZenML Service Connectors. Use ZenML Secrets to securely store and reference your Alibaba Cloud credentials. +{% endhint %} + {% tabs %} {% tab title="Using a ZenML Secret (recommended)" %} diff --git a/docs/book/component-guide/artifact-stores/minio.md b/docs/book/component-guide/artifact-stores/minio.md index 2468211cd59..d7487a634da 100644 --- a/docs/book/component-guide/artifact-stores/minio.md +++ b/docs/book/component-guide/artifact-stores/minio.md @@ -10,10 +10,11 @@ description: Storing artifacts in MinIO object storage. You should use the MinIO Artifact Store when: -* You want to run a self-hosted object storage solution -* You need S3-compatible storage in your own data center or private cloud -* You're running ZenML in a Kubernetes environment and want co-located storage -* You want to avoid cloud vendor lock-in while maintaining S3 API compatibility +* You require self-hosted object storage for data sovereignty or compliance requirements +* Your MLOps infrastructure runs on-premises or in a private cloud environment +* You need S3-compatible storage co-located with your Kubernetes-based ZenML deployment +* You want to eliminate cloud vendor dependencies while maintaining S3 API compatibility +* You're developing locally and need a lightweight S3-compatible storage backend for testing ### How do you deploy it? From c1f22d2dc30010f70c6f28b4a130713b3df7a93d Mon Sep 17 00:00:00 2001 From: Safoine El khabich Date: Tue, 9 Dec 2025 17:25:12 +0100 Subject: [PATCH 5/5] add mnio mentaince --- docs/book/component-guide/artifact-stores/minio.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/book/component-guide/artifact-stores/minio.md b/docs/book/component-guide/artifact-stores/minio.md index d7487a634da..c2def32bc56 100644 --- a/docs/book/component-guide/artifact-stores/minio.md +++ b/docs/book/component-guide/artifact-stores/minio.md @@ -6,6 +6,10 @@ description: Storing artifacts in MinIO object storage. [MinIO](https://min.io/) is a high-performance, S3-compatible object storage system. Since MinIO provides a fully S3-compatible API, you can use ZenML's S3 Artifact Store integration to connect to MinIO. +{% hint style="warning" %} +**Maintenance Mode**: The open-source MinIO project is currently in maintenance mode and is not accepting new changes. Only critical security fixes may be evaluated on a case-by-case basis. For development and testing purposes, MinIO remains a viable option, but for production use cases requiring active support, consider [MinIO AIStor](https://min.io/product/aistor) or alternative S3-compatible storage solutions like [Ceph RGW](https://ceph.io/en/discover/technology/#object). +{% endhint %} + ### When would you want to use it? You should use the MinIO Artifact Store when: