Skip to content

feat(spark): implement Spark datetime function last_day #16828

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Jul 26, 2025

Conversation

Standing-Man
Copy link
Contributor

@Standing-Man Standing-Man commented Jul 20, 2025

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Implement Spark datetime function last_day

Are these changes tested?

I added tests for last_day function.

Are there any user-facing changes?

yes, new function.

@github-actions github-actions bot added sqllogictest SQL Logic Tests (.slt) spark labels Jul 20, 2025
@Standing-Man
Copy link
Contributor Author

Hi @alamb, I’ve added the last_day function. However, running cargo test --test sqllogictests -- spark produces some errors. I’m looking into it, but please let me know if you have any insights.

1. query failed: DataFusion error: Error during planning: Invalid function 'last_day'.
Did you mean 'list_cat'?

@alamb
Copy link
Contributor

alamb commented Jul 21, 2025

Hi @alamb, I’ve added the last_day function. However, running cargo test --test sqllogictests -- spark produces some errors. I’m looking into it, but please let me know if you have any insights.

1. query failed: DataFusion error: Error during planning: Invalid function 'last_day'.
Did you mean 'list_cat'?

I think the issue is that the function is named spark_last_day in the PR (rather than last_day)

    fn name(&self) -> &str {
        "spark_last_day"
    }

Copy link
Contributor

@2010YOUY01 2010YOUY01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, looks good!

Signed-off-by: Alan Tang <jmtangcs@gmail.com>
Signed-off-by: Alan Tang <jmtangcs@gmail.com>
Signed-off-by: Alan Tang <jmtangcs@gmail.com>
Signed-off-by: Alan Tang <jmtangcs@gmail.com>
Signed-off-by: Alan Tang <jmtangcs@gmail.com>
Signed-off-by: Alan Tang <jmtangcs@gmail.com>
Signed-off-by: Alan Tang <jmtangcs@gmail.com>
@2010YOUY01 2010YOUY01 merged commit 2ff02b2 into apache:main Jul 26, 2025
29 checks passed
@alamb
Copy link
Contributor

alamb commented Jul 26, 2025

giphy

adriangb pushed a commit to pydantic/datafusion that referenced this pull request Jul 28, 2025
* feat(spark): implement Spark datetime function last_day

Signed-off-by: Alan Tang <jmtangcs@gmail.com>

* chore: fix the export function name

Signed-off-by: Alan Tang <jmtangcs@gmail.com>

* chore: Fix Cargo.toml formatting

Signed-off-by: Alan Tang <jmtangcs@gmail.com>

* test: add more tests for spark function last_day

Signed-off-by: Alan Tang <jmtangcs@gmail.com>

* feat(spark): set the signature to be taking exactly one Date32 type

Signed-off-by: Alan Tang <jmtangcs@gmail.com>

* test(spark): add more bad cases

Signed-off-by: Alan Tang <jmtangcs@gmail.com>

* chore: clean up redundant package

Signed-off-by: Alan Tang <jmtangcs@gmail.com>

---------

Signed-off-by: Alan Tang <jmtangcs@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
spark sqllogictest SQL Logic Tests (.slt)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[datafusion-spark] Implement Spark datetime function last_day
3 participants