Skip to content

wip struct field pushdown parquet #18966

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
adriangb wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

wip struct field pushdown parquet #18966

adriangb wants to merge 2 commits into from

Conversation

@adriangb
Copy link
Contributor

@adriangb adriangb commented Nov 27, 2025

Next step in #14993

@adriangb
wip struct field pushdown parquet
406722b 
need to figure out what to do with SchemaAdapter; there's a couple failing tests

Should we do this transformation before or after ExprAdapter?
@github-actions github-actions bot added core Core DataFusion crate datasource Changes to the datasource crate labels Nov 27, 2025
@adriangb adriangb mentioned this pull request Nov 27, 2025
Open
@adriangb
Copy link
Contributor Author

adriangb commented Nov 30, 2025

I have a better next step in #18998.

Essentially we migrate ParquetOpener to use PhysicalExprAdapterFactory fully and then all we have left is to implement the specialization for struct pushdown as a separate self-contained change.

@adriangb adriangb mentioned this pull request Nov 30, 2025
Merged
github-merge-queue bot pushed a commit that referenced this pull request Dec 5, 2025
@adriangb
Cut Parquet over to PhysicalExprAdapter, remove SchemaAdapter (#1…
e01fa0c 
…8998)

Chips away at #14993 and
#16800

Changes made in this PR:
- Update DefaultPhysicalExprAdapter to handle struct column evolution
- Remove use of SchemaAdapter from row filter / predicate pushdown /
late materialization. I believe it was already not doing much here other
than where tests checked specific behavior and had not been updated to
use PhysicalExprAdapter
- Changed projection handling to use `PhysicalExprAdapter` instead of
`SchemaAdapter`
- Kept intermediary `Vec<usize>` so we can use `ProjectionMask::roots`
and punt the complexity of implementing `ProjectionExprs` ->
`ProjectionMask` until a later PR (there is a draft in #18966 of what
that might look like).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

core Core DataFusion crate datasource Changes to the datasource crate

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@adriangb