Avoid dependency pinning

[lucargir]

Would it be possible to drop the dependency pinning and use a range of versions instead? This is the practice that packages such as dagster-dlt use, and given that SQLMesh makes releases quite often, I don't think it's a good idea to lock the version, unless there's some specific concern...

[ravenac95]

Unfortunately, with how fast sqlmesh moves, we must pin dependencies, we are relying on things that are internal to sqlmesh and as such this integration is a bit of a hack in general. Every minor change in sqlmesh recently has caused something to break and we aren't currently able to control that as we are using internals in sqlmesh that aren't intended to be relied upon. So to currently ensure stability for it as an integration on our side by pinning. It's not currently possible to not pin it based on the changes between the minor versions in sqlmesh. Apologies, this is sadly a "wontfix" for now.

[ravenac95]

Well I would actually note, it's less a wontfix and more a, this is the situation right now and until we can come up with a better way to deal with it we won't fix it. If you feel it's high priority enough we'd definitely accept any PRs.

[ravenac95]

Funny enough, @lucargir, I just made a PR that might be able to relieve much of the need to pin. However, it's not perfect so I'm keeping the pinning for now, as that's the safest, but I have updated to 0.174.2.

[lucargir]

@ravenac95 As long as it's possible to get eventual updates (once a month or so) as long as there are no changes that affect the integration, would be great. SQLMesh and SQLGlot change quickly, yes, but a great part of it it's because they fix issues that can affect existing integrations, so for me it's important at least to have the possibility of requesting a quick dep check and upgrading if this is the case. Of course, if an upgrade requires code changes, it's reasonable that it's paused until a solution can be worked on, but then I would also be happy to contribute with a PR. I'll work on #32 next week and propose a PR.

[ravenac95]

@lucargir reopening this now. I think with the changes I've made this might be achievable but some better testing and publishing infrastructure should be setup. Instead of pinning I think we should have a set of ranges.

Just a note, the latest upgrade seems to have worked without issue based on the available tests. I think the thing to do is something like the following:

• Periodically run tests to upgrade sqlmesh, if it passes the automation, this should create a PR (or automagically update).
  • This should ideally create the range in pyproject.toml. It would prevent folks from installing this and things breaking, but the new updates to the library do attempt to detect incompatibilities in the installed sqlmesh.
• Starting perhaps from the most recent pinned version (0.178.x), we use that as the starting range of compatibility.

[lucargir]

@ravenac95 I think this could be done with dependabot and the current PR tests, right? As in, dependabot creates a PR to upgrade the deps (updating the upper limit), the tests get triggered, and then you'd just need to approve and merge.

[geoHeil]

Any updates on this by now?

[lucargir]

@geoHeil There's PR #49 in progress to make some changes, and iirc they mentioned just having a minimum version of SQLMesh.

[ravenac95]

@geoHeil @lucargir that's correct, the solution in #49 is alright but I realize there might be a better way that won't be as brittle even as the current solution. I'm hoping to get something done soon here.

Additionally, the current solution uses a bit of a hacky exec to satisfy the console changes, but I think it's better to use a generic callable class.

[ravenac95]

finally with #52 merged I think it's possible to potentially remove pinning.