Add Docker-based Ceph + Polaris cluster setup

[sharas2050]

This PR introduces a complete Docker Compose–based Ceph cluster environment for Polaris integration.
The goal is to make it easy to simplify the developer experience for Polaris contributors and testers who want to:

Experiment with Ceph-based S3 storage locally — including MON, MGR, OSD, RGW
Validate Polaris integration against RGW without a full cluster deployment

• Added initialization flow for Ceph monitor and manager daemons
• Added OSD creation
• Added preconfigured RGW service with S3-compatible endpoint and pre-created user
• Added Polaris bootstrap and connection to Ceph RGW for object storage testing
• note: This setup runs without STS for simplicity (manual key/secret configuration)

[snazy]

Just out of curiosity, @sharas2050 do you know how much (or better: how many containers) would be minimally needed to run Ceph with STS/IAM?

[sharas2050]

Just out of curiosity, @sharas2050 do you know how much (or better: how many containers) would be minimally needed to run Ceph with STS/IAM?

It is the same number as using it without STS. It is matter of Ceph configuration and bucket management. I wrote a small article earlier this year about that. You can read it here. Even in this example You can combine MON+MGR in a single container.

[dimas-b]

Thanks for your contribution, @sharas2050 ! The LGTM overall, just a minor comments about .env.

[snazy]

Would you mind adding a getting-started with IAM/STS as a follow-up of this PR?

[sharas2050]

@snazy I just updated PR with your suggested changes

[snazy]

@snazy I just updated PR with your suggested changes

Let me try it locally.

[dimas-b]

Sadly, mon1 fails in my env. (Linux + Podman)... logs:

+ sudo -u ceph ceph-mon --mkfs -i mon1 --monmap /var/lib/ceph/tmp/monmap --keyring /var/lib/ceph/tmp/ceph.mon.keyring
sudo: PAM account management error: Authentication service cannot retrieve authentication info
sudo: a password is required

[snazy]

There's a new failure:

$ docker compose up -d mon1 mgr

[+] Running 0/1
 ⠹ Container ceph-mon1-1  Starting                                                                                                                                                                                                                                                                     0.2s 
Error response from daemon: crun: cannot stat `.../polaris/getting-started/ceph/bootstrap-osd`: No such file or directory: OCI runtime attempted to invoke a command that was not found
Error: executing /usr/libexec/docker/cli-plugins/docker-compose up -d mon1 mgr: exit status 1

Could you cross check that all steps work from a clean environment?

[sharas2050]

There's a new failure:

$ docker compose up -d mon1 mgr

[+] Running 0/1
 ⠹ Container ceph-mon1-1  Starting                                                                                                                                                                                                                                                                     0.2s 
Error response from daemon: crun: cannot stat `.../polaris/getting-started/ceph/bootstrap-osd`: No such file or directory: OCI runtime attempted to invoke a command that was not found
Error: executing /usr/libexec/docker/cli-plugins/docker-compose up -d mon1 mgr: exit status 1

Could you cross check that all steps work from a clean environment?

The issue is that Podman doesn't auto-create the directory on the host like Docker does. Initialized directory in container instead.