Skip to content

Complete Fetch Phase (EXTERNAL_LINKS disposition + ARROW format) #596

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
varun-edachali-dbx wants to merge 388 commits into from
Closed

Complete Fetch Phase (EXTERNAL_LINKS disposition + ARROW format) #596

varun-edachali-dbx wants to merge 388 commits into from

Conversation

@varun-edachali-dbx
Copy link
Contributor

@varun-edachali-dbx varun-edachali-dbx commented Jun 13, 2025 •
edited
Loading

WIP

What type of PR is this?

  • Refactor
  • Feature
  • Bug Fix
  • Other

Description

How is this tested?

  • Unit tests
  • E2E Tests
  • Manually
  • N/A

Related Tickets & Documents

varun-edachali-dbx and others added 30 commits May 28, 2025 03:49
@varun-edachali-dbx
fix: Any is not defined
7461715 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
fix: get_session_id_hex() is not defined
b295acd 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
command_handle -> command_id in ExecuteResponse
8afc5d5 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
fix: active op handle -> active command id in Cursor
b8f9146 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
fixed (most) tests by accounting for normalised Session interface
7c733ee 
have not considered how to fix the protocol version check yet

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
fix: convert command id to operationHandle in status_request
0917ea1 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
decouple session class from existing Connection
3fd2a46 
ensure maintenance of current APIs of Connection while delegating
responsibility

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
add open property to Connection to ensure maintenance of existing API
03d3ae7 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
use connection open property instead of long chain through session
fb0fa46 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
trigger integration workflow
e3770cd 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
fix: ensure open attribute of Connection never fails
3b8002f 
in case the openSession takes long, the initialisation of the session
will not complete immediately. This could make the session attribute
inaccessible. If the Connection is deleted in this time, the open()
check will throw because the session attribute does not exist. Thus, we
default to the Connection being closed in this case. This was not an
issue before because open was a direct attribute of the Connection
class. Caught in the integration tests.

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
fix: de-complicate earlier connection open logic
f1a350a 
earlier, one of the integration tests was failing because 'session was
not an attribute of Connection'. This is likely tied to a local
configuration issue related to unittest that was causing an error in the
test suite itself. The tests are now passing without checking for the
session attribute.
c676f9b

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
Revert "fix: de-complicate earlier connection open logic"
98b0dc7 
This reverts commit d6b1b19.

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
[empty commit] attempt to trigger ci e2e workflow
afee423 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@saishreeeee @varun-edachali-dbx
PECOBLR-86 improve logging on python driver (#556)
2d24fdd 
* PECOBLR-86 Improve logging for debug level

Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>

* PECOBLR-86 Improve logging for debug level

Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>

* fixed format

Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>

* used lazy logging

Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>

* changed debug to error logs

Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>

* used lazy logging

Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>

---------

Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
Revert "Merge remote-tracking branch 'upstream/sea-migration' into de…
a5561e8 
…couple-session"

This reverts commit dbb2ec5, reversing
changes made to 7192f11.

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
Reapply "Merge remote-tracking branch 'upstream/sea-migration' into d…
0d890a5 
…ecouple-session"

This reverts commit bdb8381.

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
fix: separate session opening logic from instantiation
c2aa762 
ensures correctness of self.session.open call in Connection

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
chore: use get_handle() instead of private session attribute in client
fe642da 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
fix: remove accidentally removed assertions
394333c 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
generalise open session, fix session tests to consider positional args
ef07acd 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
formatting (black)
1ef46cf 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
correct session logic after duplication during merge
76ca997 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
args -> kwargs in tests
afc6f8f 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
delegate protocol version to SessionId
f6660ba 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
ids -> backend/types
9871a93 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
update open session with normalised SessionId
595d795 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
remove merge artifacts, account for result set
10ee940 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
fix: import CommandId in client tests
3d75d6c
@varun-edachali-dbx
expect session_id in protocol version getter
b8e1bbd 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@github-actions
Copy link

github-actions bot commented Jun 13, 2025

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

@varun-edachali-dbx
Revert "merge"
0ae2ab0 
This reverts commit c0bf461, reversing
changes made to 75505cf.
@github-actions
Copy link

github-actions bot commented Jun 13, 2025

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

@varun-edachali-dbx
remove duplicate link addition
09e8b83 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@github-actions
Copy link

github-actions bot commented Jun 16, 2025

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

@varun-edachali-dbx
remove download manager changes
c43dbce 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@github-actions
Copy link

github-actions bot commented Jun 16, 2025

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

@varun-edachali-dbx
add add_links func
0fc79ec 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@github-actions
Copy link

github-actions bot commented Jun 16, 2025

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

@varun-edachali-dbx
remove excess changes
8257529 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@github-actions
Copy link

github-actions bot commented Jun 16, 2025

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

@varun-edachali-dbx
remove excess methods
c372788 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@github-actions
Copy link

github-actions bot commented Jun 16, 2025

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

@varun-edachali-dbx
correct get chunk info endpoint
75d5bc1 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@github-actions
Copy link

github-actions bot commented Jun 16, 2025

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

@varun-edachali-dbx
simplify remaining_rows fetch
9ddb3ec 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
reduce repetition in cloud fetch queue
58d1d7f 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@github-actions
Copy link

github-actions bot commented Jun 16, 2025

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

@varun-edachali-dbx
Copy link
Contributor Author

varun-edachali-dbx commented Jun 16, 2025

Superseded by #598

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@deeksha-db deeksha-db Awaiting requested review from deeksha-db deeksha-db will be requested when the pull request is marked ready for review deeksha-db is a code owner

@samikshya-db samikshya-db Awaiting requested review from samikshya-db samikshya-db will be requested when the pull request is marked ready for review samikshya-db is a code owner

@jprakash-db jprakash-db Awaiting requested review from jprakash-db jprakash-db will be requested when the pull request is marked ready for review jprakash-db is a code owner

@jackyhu-db jackyhu-db Awaiting requested review from jackyhu-db jackyhu-db will be requested when the pull request is marked ready for review jackyhu-db is a code owner

@madhav-db madhav-db Awaiting requested review from madhav-db madhav-db will be requested when the pull request is marked ready for review madhav-db is a code owner

@gopalldb gopalldb Awaiting requested review from gopalldb gopalldb will be requested when the pull request is marked ready for review gopalldb is a code owner

@jayantsing-db jayantsing-db Awaiting requested review from jayantsing-db jayantsing-db will be requested when the pull request is marked ready for review jayantsing-db is a code owner

@vikrantpuppala vikrantpuppala Awaiting requested review from vikrantpuppala vikrantpuppala will be requested when the pull request is marked ready for review vikrantpuppala is a code owner

@shivam2680 shivam2680 Awaiting requested review from shivam2680 shivam2680 will be requested when the pull request is marked ready for review shivam2680 is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.