Skip to content

Conversation

jameslamb
Copy link
Owner

@jameslamb jameslamb commented Dec 31, 2024

More miscellaneous refactoring.

Notes for Reviewers

Why set CPU architecture?

This repo only builds single-architecture container images.
The notebooks are intended to be run locally, from the same machine where you built all the images.

dask_cloudprovider.FargateCluster defaults to x86_64.
When I built the images on my M2 Mac (which is arm64) then tried to run the demo-aws notebook as-is, I saw this error on ECS:

exec /usr/local/bin/dask-scheduler: exec format error

This fixes that by detecting CPU architecture on the machine where the notebook is running, and passing that through. Thanks to @dmitry-livchak for adding that support in dask/dask-cloudprovider#425 😁

Why make LightGBM/ read-write at build time?

Without it, the build fails like this:

cp: cannot create directory './lightgbm-python': Read-only file system

Because build-python.sh from LightGBM creates a temporary directory and moves files into it (code link).

@jameslamb jameslamb changed the title build profiling image in CI build profiling image in CI, add arm64 support to AWS notebooks Dec 31, 2024
@jameslamb jameslamb merged commit 59250f3 into main Dec 31, 2024
3 checks passed
@jameslamb jameslamb deleted the more-refactoring branch December 31, 2024 04:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant