-
Notifications
You must be signed in to change notification settings - Fork 377
Add support for UV dependency management #2601
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks great @geruh, I hear a lot of good stuff around uv
🙌
"Programming Language :: Python :: 3.12", | ||
"License :: OSI Approved :: Apache Software License", | ||
"Operating System :: OS Independent", | ||
"Programming Language :: Python :: 3.9", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice one, maybe we should add pretty-format-toml
to pre-commit to format the toml
files 👍
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's a great idea. #2605 if you want to merge it in
|
||
[project.optional-dependencies] | ||
pyarrow = [ | ||
"pyarrow>=17.0.0", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit, maybe find a way to deduplicate the constraints. pyarrow
is mentioned multiple times.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sounds good! it looks like poetry had a way of doing this before with the [tool.poetry.extras]
, kind of acting as a version catalog. UV doesn't have support for this but respects the PEP standard for project dependencies. In this case one way we can de-duplicate is by using the PEP 508 Extras grammar which allows for self referencing other dependencies in the project like:
[project.optional-dependencies]
pyarrow = [
"pyarrow>=17.0.0",
"pyiceberg-core>=0.5.1,<0.7.0",
]
pandas = [
"pyiceberg[pyarrow]",
"pandas>=1.0.0,<3.0.0",
]
What do you think @Fokko?
Closes #2553
Rationale for this change
During the release process cleanup, I also had prototyped a migration from Poetry to UV. I used the https://github.com/mkniewallner/migrate-to-uv package to migrate to UV and ran a release workflow and some others like doc serve for validation.
This PR also migrates the Makefile to use UV. I'll still need to upgrade the docs to reflect the UV contribution workflow. I think it would be a good start to push up as a prototype, and have others checkout and give it a try.
Build System Changes
Build Backend
UV doesn't have a build backend like poetry that supports hooks or custom build scripts. So I've switched to use Hatchling as the build backend with the existing (
build-module.py
) to ensure Avro files are properly compiled and included.Build Verification
Release builds can be verified using:
Both commands ensure parity with the previous Poetry build outputs.
Are these changes tested?
Yes
make install
Environment setup and dependency installationmake test
- Unit testsmake test-integration
- Integration tests with all extrasmake lint
- Linting and code quality checksmake doc-install && make doc-serve
- DocumentationAre there any user-facing changes?
No user facing changes. Contributors will need to use UV instead of Poetry for development, but all make targets remain the same. So not really?