-
Notifications
You must be signed in to change notification settings - Fork 559
Add Alibaba Cloud storage artifact-store #4289
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
safoinme
wants to merge
6
commits into
develop
Choose a base branch
from
docs/add-alibaba-minio-docs
base: develop
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+190
−6
Open
Changes from 1 commit
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
b195cbb
init
safoinme e3fe974
update docs
safoinme 0612488
Merge branch 'develop' into docs/add-alibaba-minio-docs
safoinme e871194
update secret namiong
safoinme bd861b4
Merge branch 'docs/add-alibaba-minio-docs' of https://github.com/zenm…
safoinme 0622f34
fixes by claude
safoinme File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,92 @@ | ||
| --- | ||
| description: Storing artifacts in Alibaba Cloud Object Storage Service (OSS). | ||
| --- | ||
|
|
||
| # Alibaba Cloud OSS | ||
|
|
||
| Alibaba Cloud Object Storage Service (OSS) is an S3-compatible object storage service. Since OSS provides an S3-compatible API, you can use ZenML's S3 Artifact Store integration to connect to Alibaba Cloud OSS. | ||
|
|
||
| ### When would you want to use it? | ||
|
|
||
| You should use the Alibaba Cloud OSS Artifact Store when you want to store your ZenML artifacts in Alibaba Cloud infrastructure, particularly if: | ||
|
|
||
| * You're already using Alibaba Cloud services | ||
| * You need to store artifacts in specific geographic regions served by Alibaba Cloud | ||
| * You want to leverage Alibaba Cloud's pricing and performance characteristics | ||
safoinme marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| ### How do you deploy it? | ||
|
|
||
| Since Alibaba Cloud OSS is S3-compatible, you'll use the S3 integration. First, install it: | ||
|
|
||
| ```shell | ||
| zenml integration install s3 -y | ||
| ``` | ||
|
|
||
| You'll also need to create an OSS bucket and obtain your access credentials from the Alibaba Cloud console. | ||
|
|
||
| ### How do you configure it? | ||
|
|
||
| To use Alibaba Cloud OSS with ZenML, you need to configure the S3 Artifact Store with specific settings for OSS compatibility: | ||
|
|
||
| {% tabs %} | ||
| {% tab title="Using a ZenML Secret (recommended)" %} | ||
safoinme marked this conversation as resolved.
Show resolved
Hide resolved
safoinme marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| First, create a ZenML secret with your Alibaba Cloud credentials: | ||
|
|
||
| ```shell | ||
| zenml secret create alibaba_secret \ | ||
| --aws_access_key_id='<YOUR_ALIBABA_ACCESS_KEY_ID>' \ | ||
| --aws_secret_access_key='<YOUR_ALIBABA_SECRET_ACCESS_KEY>' | ||
safoinme marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| ``` | ||
safoinme marked this conversation as resolved.
Show resolved
Hide resolved
safoinme marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| Then register the artifact store with the required OSS configuration: | ||
|
|
||
| ```shell | ||
| zenml artifact-store register alibaba_store -f s3 \ | ||
| --path='s3://your-bucket-name' \ | ||
| --authentication_secret=alibaba_secret \ | ||
| --client_kwargs='{"endpoint_url": "https://oss-<region>.aliyuncs.com"}' \ | ||
| --config_kwargs='{"signature_version": "s3", "s3": {"addressing_style": "virtual"}}' | ||
safoinme marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| ``` | ||
| {% endtab %} | ||
|
|
||
| {% tab title="Inline credentials" %} | ||
|
|
||
| You can also provide credentials directly (not recommended for production): | ||
|
|
||
| ```shell | ||
| zenml artifact-store register alibaba_store -f s3 \ | ||
| --path='s3://your-bucket-name' \ | ||
| --aws_access_key_id='<YOUR_ALIBABA_ACCESS_KEY_ID>' \ | ||
| --aws_secret_access_key='<YOUR_ALIBABA_SECRET_ACCESS_KEY>' \ | ||
| --client_kwargs='{"endpoint_url": "https://oss-<region>.aliyuncs.com"}' \ | ||
| --config_kwargs='{"signature_version": "s3", "s3": {"addressing_style": "virtual"}}' | ||
| ``` | ||
| {% endtab %} | ||
| {% endtabs %} | ||
safoinme marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| {% hint style="warning" %} | ||
| **Important:** When using Alibaba Cloud OSS, you must set the following `config_kwargs`: | ||
|
|
||
| ```json | ||
| {"signature_version": "s3", "s3": {"addressing_style": "virtual"}} | ||
| ``` | ||
|
|
||
| This is required for proper compatibility with Alibaba Cloud OSS's S3 API implementation. | ||
| {% endhint %} | ||
|
|
||
| Replace `<region>` with your OSS region (e.g., `eu-central-1`, `cn-hangzhou`, `ap-southeast-1`). You can find the list of available regions and their endpoints in the [Alibaba Cloud OSS documentation](https://www.alibabacloud.com/help/en/oss/user-guide/regions-and-endpoints). | ||
|
|
||
| Finally, add the artifact store to your stack: | ||
|
|
||
| ```shell | ||
| zenml stack register custom_stack -a alibaba_store ... --set | ||
safoinme marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| ``` | ||
|
|
||
| ### How do you use it? | ||
|
|
||
| Using the Alibaba Cloud OSS Artifact Store is no different from [using any other flavor of Artifact Store](./#how-to-use-it). ZenML handles the S3-compatible API translation automatically. | ||
|
|
||
| For more details on the S3 Artifact Store configuration options, refer to the [S3 Artifact Store documentation](s3.md). | ||
|
|
||
| <figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure> | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,89 @@ | ||
| --- | ||
| description: Storing artifacts in MinIO object storage. | ||
| --- | ||
|
|
||
| # MinIO | ||
|
|
||
| [MinIO](https://min.io/) is a high-performance, S3-compatible object storage system. Since MinIO provides a fully S3-compatible API, you can use ZenML's S3 Artifact Store integration to connect to MinIO. | ||
|
|
||
| ### When would you want to use it? | ||
|
|
||
| You should use the MinIO Artifact Store when: | ||
|
|
||
| * You want to run a self-hosted object storage solution | ||
| * You need S3-compatible storage in your own data center or private cloud | ||
| * You're running ZenML in a Kubernetes environment and want co-located storage | ||
| * You want to avoid cloud vendor lock-in while maintaining S3 API compatibility | ||
safoinme marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| ### How do you deploy it? | ||
|
|
||
| Since MinIO is S3-compatible, you'll use the S3 integration. First, install it: | ||
|
|
||
| ```shell | ||
| zenml integration install s3 -y | ||
| ``` | ||
|
|
||
| You'll also need a running MinIO instance. MinIO can be deployed in various ways: | ||
|
|
||
| * **Docker**: `docker run -p 9000:9000 -p 9001:9001 minio/minio server /data --console-address ":9001"` | ||
| * **Kubernetes**: Using the [MinIO Operator](https://min.io/docs/minio/kubernetes/upstream/) | ||
| * **Binary**: Download from [MinIO's website](https://min.io/download) | ||
safoinme marked this conversation as resolved.
Show resolved
Hide resolved
safoinme marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| ### How do you configure it? | ||
|
|
||
| To use MinIO with ZenML, configure the S3 Artifact Store with your MinIO endpoint: | ||
|
|
||
| {% tabs %} | ||
| {% tab title="Using a ZenML Secret (recommended)" %} | ||
|
|
||
| First, create a ZenML secret with your MinIO credentials: | ||
|
|
||
| ```shell | ||
| zenml secret create minio_secret \ | ||
| --aws_access_key_id='<YOUR_MINIO_ACCESS_KEY>' \ | ||
| --aws_secret_access_key='<YOUR_MINIO_SECRET_KEY>' | ||
safoinme marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| ``` | ||
safoinme marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| Then register the artifact store: | ||
|
|
||
| ```shell | ||
| zenml artifact-store register minio_store -f s3 \ | ||
| --path='s3://your-bucket-name' \ | ||
| --authentication_secret=minio_secret \ | ||
| --client_kwargs='{"endpoint_url": "http://minio.example.com:9000"}' | ||
| ``` | ||
| {% endtab %} | ||
|
|
||
| {% tab title="Inline credentials" %} | ||
|
|
||
| You can also provide credentials directly (not recommended for production): | ||
|
|
||
| ```shell | ||
| zenml artifact-store register minio_store -f s3 \ | ||
| --path='s3://your-bucket-name' \ | ||
| --aws_access_key_id='<YOUR_MINIO_ACCESS_KEY>' \ | ||
| --aws_secret_access_key='<YOUR_MINIO_SECRET_KEY>' \ | ||
| --client_kwargs='{"endpoint_url": "http://minio.example.com:9000"}' | ||
| ``` | ||
| {% endtab %} | ||
| {% endtabs %} | ||
|
|
||
| Replace `http://minio.example.com:9000` with your actual MinIO endpoint. If you're running MinIO locally for development, this might be `http://localhost:9000`. | ||
|
|
||
| {% hint style="info" %} | ||
| If your MinIO instance uses HTTPS with a self-signed certificate, you may need to configure SSL verification. Consult the [S3 Artifact Store documentation](s3.md#advanced-configuration) for advanced configuration options. | ||
| {% endhint %} | ||
|
|
||
| Finally, add the artifact store to your stack: | ||
|
|
||
| ```shell | ||
| zenml stack register custom_stack -a minio_store ... --set | ||
safoinme marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| ``` | ||
|
|
||
| ### How do you use it? | ||
|
|
||
| Using the MinIO Artifact Store is no different from [using any other flavor of Artifact Store](./#how-to-use-it). ZenML handles the S3-compatible API translation automatically. | ||
|
|
||
| For more details on the S3 Artifact Store configuration options, refer to the [S3 Artifact Store documentation](s3.md). | ||
|
|
||
| <figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure> | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.