Skip to content

Conversation

@Dandandan
Copy link
Contributor

@Dandandan Dandandan commented Nov 27, 2025

Which issue does this PR close?

Rationale for this change

Improves the plan / plan readability for queries with joins.

Seems to improve perf as well a bit for more challenging joins: (TPC-H SF=10, in-memory)

│ QQuery 21    │              7948.54 ms │    6340.74 ms │ +1.25x faster │
│ QQuery 22    │                58.30 ms │      49.86 ms │ +1.17x faster │

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

@github-actions github-actions bot added optimizer Optimizer rules sqllogictest SQL Logic Tests (.slt) physical-plan Changes to the physical-plan crate labels Nov 27, 2025
@Dandandan Dandandan changed the title Refactor join2 Coalesce batches inside hash join Nov 27, 2025
@github-actions github-actions bot added the core Core DataFusion crate label Nov 27, 2025
@Dandandan Dandandan changed the title Coalesce batches inside hash join Coalesce batches inside hash join, reuse indices buffer Nov 28, 2025
@Dandandan
Copy link
Contributor Author

@alamb let's fire some benchmarks?
I mostly see improvement on SF=10 in memory on the standard set of benchmarks

@alamb
Copy link
Contributor

alamb commented Nov 29, 2025

🤖 ./gh_compare_branch.sh Benchmark Script Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing refactor_join2 (60bdf21) to 02a7181 diff using: tpch10
Results will be posted here when complete

@alamb
Copy link
Contributor

alamb commented Nov 29, 2025

@alamb let's fire some benchmarks? I mostly see improvement on SF=10 in memory on the standard set of benchmarks

I started some. It would be really nice to find some way for others to run their own -- I don't think it is great I am often the bottleneck.

Maybe I'll see if I can get an LLM to write me a script to do that...

@alamb
Copy link
Contributor

alamb commented Nov 30, 2025

run benchmarks

@alamb
Copy link
Contributor

alamb commented Nov 30, 2025

🤖 ./gh_compare_branch.sh Benchmark Script Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing refactor_join2 (60bdf21) to 02a7181 diff using: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete

@alamb
Copy link
Contributor

alamb commented Dec 1, 2025

🤖: Benchmark completed

Details

Comparing HEAD and refactor_join2
--------------------
Benchmark tpch_sf10.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Query        ┃ HEAD ┃ refactor_join2 ┃       Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ QQuery 1     │ FAIL │           FAIL │ incomparable │
│ QQuery 2     │ FAIL │           FAIL │ incomparable │
│ QQuery 3     │ FAIL │           FAIL │ incomparable │
│ QQuery 4     │ FAIL │           FAIL │ incomparable │
│ QQuery 5     │ FAIL │           FAIL │ incomparable │
│ QQuery 6     │ FAIL │           FAIL │ incomparable │
│ QQuery 7     │ FAIL │           FAIL │ incomparable │
│ QQuery 8     │ FAIL │           FAIL │ incomparable │
│ QQuery 9     │ FAIL │           FAIL │ incomparable │
│ QQuery 10    │ FAIL │           FAIL │ incomparable │
│ QQuery 11    │ FAIL │           FAIL │ incomparable │
│ QQuery 12    │ FAIL │           FAIL │ incomparable │
│ QQuery 13    │ FAIL │           FAIL │ incomparable │
│ QQuery 14    │ FAIL │           FAIL │ incomparable │
│ QQuery 15    │ FAIL │           FAIL │ incomparable │
│ QQuery 16    │ FAIL │           FAIL │ incomparable │
│ QQuery 17    │ FAIL │           FAIL │ incomparable │
│ QQuery 18    │ FAIL │           FAIL │ incomparable │
│ QQuery 19    │ FAIL │           FAIL │ incomparable │
│ QQuery 20    │ FAIL │           FAIL │ incomparable │
│ QQuery 21    │ FAIL │           FAIL │ incomparable │
│ QQuery 22    │ FAIL │           FAIL │ incomparable │
└──────────────┴──────┴────────────────┴──────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┓
┃ Benchmark Summary             ┃        ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━┩
│ Total Time (HEAD)             │ 0.00ms │
│ Total Time (refactor_join2)   │ 0.00ms │
│ Average Time (HEAD)           │ 0.00ms │
│ Average Time (refactor_join2) │ 0.00ms │
│ Queries Faster                │      0 │
│ Queries Slower                │      0 │
│ Queries with No Change        │      0 │
│ Queries with Failure          │     22 │
└───────────────────────────────┴────────┘

@Dandandan
Copy link
Contributor Author

Looks like the tpch_mem10 crashed it @alamb ;)

@Dandandan
Copy link
Contributor Author

run benchmark tpch_mem

@Dandandan
Copy link
Contributor Author

run benchmark tpch10

@alamb
Copy link
Contributor

alamb commented Dec 1, 2025

🤖 ./gh_compare_branch.sh Benchmark Script Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing refactor_join2 (60bdf21) to 02a7181 diff using: tpch_mem
Results will be posted here when complete

@alamb
Copy link
Contributor

alamb commented Dec 1, 2025

🤖: Benchmark completed

Details

Comparing HEAD and refactor_join2
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃      HEAD ┃ refactor_join2 ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1     │ 134.66 ms │      136.91 ms │     no change │
│ QQuery 2     │  26.18 ms │       27.41 ms │     no change │
│ QQuery 3     │  38.45 ms │       34.18 ms │ +1.12x faster │
│ QQuery 4     │  28.86 ms │       28.26 ms │     no change │
│ QQuery 5     │  86.16 ms │       88.33 ms │     no change │
│ QQuery 6     │  19.41 ms │       19.70 ms │     no change │
│ QQuery 7     │ 219.76 ms │      225.05 ms │     no change │
│ QQuery 8     │  32.41 ms │       34.05 ms │  1.05x slower │
│ QQuery 9     │ 103.75 ms │      103.01 ms │     no change │
│ QQuery 10    │  64.61 ms │       65.53 ms │     no change │
│ QQuery 11    │  16.67 ms │       18.39 ms │  1.10x slower │
│ QQuery 12    │  51.68 ms │       52.83 ms │     no change │
│ QQuery 13    │  47.11 ms │       47.29 ms │     no change │
│ QQuery 14    │  14.15 ms │       13.86 ms │     no change │
│ QQuery 15    │  24.26 ms │       25.04 ms │     no change │
│ QQuery 16    │  25.20 ms │       25.12 ms │     no change │
│ QQuery 17    │ 146.69 ms │      151.91 ms │     no change │
│ QQuery 18    │ 280.18 ms │      283.13 ms │     no change │
│ QQuery 19    │  38.16 ms │       49.43 ms │  1.30x slower │
│ QQuery 20    │  48.97 ms │       61.59 ms │  1.26x slower │
│ QQuery 21    │ 319.97 ms │      323.01 ms │     no change │
│ QQuery 22    │  17.98 ms │       18.17 ms │     no change │
└──────────────┴───────────┴────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary             ┃           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD)             │ 1785.27ms │
│ Total Time (refactor_join2)   │ 1832.20ms │
│ Average Time (HEAD)           │   81.15ms │
│ Average Time (refactor_join2) │   83.28ms │
│ Queries Faster                │         1 │
│ Queries Slower                │         4 │
│ Queries with No Change        │        17 │
│ Queries with Failure          │         0 │
└───────────────────────────────┴───────────┘

@alamb
Copy link
Contributor

alamb commented Dec 1, 2025

🤖 ./gh_compare_branch.sh Benchmark Script Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing refactor_join2 (60bdf21) to 02a7181 diff using: tpch10
Results will be posted here when complete

@alamb
Copy link
Contributor

alamb commented Dec 1, 2025

🤖: Benchmark completed

Details

Comparing HEAD and refactor_join2
--------------------
Benchmark tpch_sf10.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Query        ┃ HEAD ┃ refactor_join2 ┃       Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ QQuery 1     │ FAIL │           FAIL │ incomparable │
│ QQuery 2     │ FAIL │           FAIL │ incomparable │
│ QQuery 3     │ FAIL │           FAIL │ incomparable │
│ QQuery 4     │ FAIL │           FAIL │ incomparable │
│ QQuery 5     │ FAIL │           FAIL │ incomparable │
│ QQuery 6     │ FAIL │           FAIL │ incomparable │
│ QQuery 7     │ FAIL │           FAIL │ incomparable │
│ QQuery 8     │ FAIL │           FAIL │ incomparable │
│ QQuery 9     │ FAIL │           FAIL │ incomparable │
│ QQuery 10    │ FAIL │           FAIL │ incomparable │
│ QQuery 11    │ FAIL │           FAIL │ incomparable │
│ QQuery 12    │ FAIL │           FAIL │ incomparable │
│ QQuery 13    │ FAIL │           FAIL │ incomparable │
│ QQuery 14    │ FAIL │           FAIL │ incomparable │
│ QQuery 15    │ FAIL │           FAIL │ incomparable │
│ QQuery 16    │ FAIL │           FAIL │ incomparable │
│ QQuery 17    │ FAIL │           FAIL │ incomparable │
│ QQuery 18    │ FAIL │           FAIL │ incomparable │
│ QQuery 19    │ FAIL │           FAIL │ incomparable │
│ QQuery 20    │ FAIL │           FAIL │ incomparable │
│ QQuery 21    │ FAIL │           FAIL │ incomparable │
│ QQuery 22    │ FAIL │           FAIL │ incomparable │
└──────────────┴──────┴────────────────┴──────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┓
┃ Benchmark Summary             ┃        ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━┩
│ Total Time (HEAD)             │ 0.00ms │
│ Total Time (refactor_join2)   │ 0.00ms │
│ Average Time (HEAD)           │ 0.00ms │
│ Average Time (refactor_join2) │ 0.00ms │
│ Queries Faster                │      0 │
│ Queries Slower                │      0 │
│ Queries with No Change        │      0 │
│ Queries with Failure          │     22 │
└───────────────────────────────┴────────┘

@alamb
Copy link
Contributor

alamb commented Dec 1, 2025

I am not sure what is going on with tpch10/tpch_mem10.

@Dandandan
Copy link
Contributor Author

run benchmark tpch10

@alamb
Copy link
Contributor

alamb commented Dec 1, 2025

🤖 ./gh_compare_branch.sh Benchmark Script Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing refactor_join2 (60bdf21) to 02a7181 diff using: tpch10
Results will be posted here when complete

@alamb
Copy link
Contributor

alamb commented Dec 1, 2025

🤖: Benchmark completed

Details

Comparing HEAD and refactor_join2
--------------------
Benchmark tpch_sf10.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Query        ┃ HEAD ┃ refactor_join2 ┃       Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ QQuery 1     │ FAIL │           FAIL │ incomparable │
│ QQuery 2     │ FAIL │           FAIL │ incomparable │
│ QQuery 3     │ FAIL │           FAIL │ incomparable │
│ QQuery 4     │ FAIL │           FAIL │ incomparable │
│ QQuery 5     │ FAIL │           FAIL │ incomparable │
│ QQuery 6     │ FAIL │           FAIL │ incomparable │
│ QQuery 7     │ FAIL │           FAIL │ incomparable │
│ QQuery 8     │ FAIL │           FAIL │ incomparable │
│ QQuery 9     │ FAIL │           FAIL │ incomparable │
│ QQuery 10    │ FAIL │           FAIL │ incomparable │
│ QQuery 11    │ FAIL │           FAIL │ incomparable │
│ QQuery 12    │ FAIL │           FAIL │ incomparable │
│ QQuery 13    │ FAIL │           FAIL │ incomparable │
│ QQuery 14    │ FAIL │           FAIL │ incomparable │
│ QQuery 15    │ FAIL │           FAIL │ incomparable │
│ QQuery 16    │ FAIL │           FAIL │ incomparable │
│ QQuery 17    │ FAIL │           FAIL │ incomparable │
│ QQuery 18    │ FAIL │           FAIL │ incomparable │
│ QQuery 19    │ FAIL │           FAIL │ incomparable │
│ QQuery 20    │ FAIL │           FAIL │ incomparable │
│ QQuery 21    │ FAIL │           FAIL │ incomparable │
│ QQuery 22    │ FAIL │           FAIL │ incomparable │
└──────────────┴──────┴────────────────┴──────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┓
┃ Benchmark Summary             ┃        ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━┩
│ Total Time (HEAD)             │ 0.00ms │
│ Total Time (refactor_join2)   │ 0.00ms │
│ Average Time (HEAD)           │ 0.00ms │
│ Average Time (refactor_join2) │ 0.00ms │
│ Queries Faster                │      0 │
│ Queries Slower                │      0 │
│ Queries with No Change        │      0 │
│ Queries with Failure          │     22 │
└───────────────────────────────┴────────┘

@Dandandan
Copy link
Contributor Author

run benchmark tpch_mem

@alamb
Copy link
Contributor

alamb commented Dec 2, 2025

🤖 ./gh_compare_branch.sh Benchmark Script Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing refactor_join2 (6bd9059) to cab28a6 diff using: tpch_mem
Results will be posted here when complete

@alamb
Copy link
Contributor

alamb commented Dec 2, 2025

🤖: Benchmark completed

Details

Comparing HEAD and refactor_join2
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃      HEAD ┃ refactor_join2 ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1     │ 140.77 ms │      133.05 ms │ +1.06x faster │
│ QQuery 2     │  29.08 ms │       27.05 ms │ +1.08x faster │
│ QQuery 3     │  37.91 ms │       38.56 ms │     no change │
│ QQuery 4     │  28.71 ms │       29.86 ms │     no change │
│ QQuery 5     │  89.24 ms │       91.30 ms │     no change │
│ QQuery 6     │  25.65 ms │       20.08 ms │ +1.28x faster │
│ QQuery 7     │ 231.33 ms │      235.99 ms │     no change │
│ QQuery 8     │  32.75 ms │       34.41 ms │  1.05x slower │
│ QQuery 9     │ 108.83 ms │      104.69 ms │     no change │
│ QQuery 10    │  63.81 ms │       64.99 ms │     no change │
│ QQuery 11    │  18.04 ms │       17.02 ms │ +1.06x faster │
│ QQuery 12    │  50.56 ms │       51.74 ms │     no change │
│ QQuery 13    │  48.63 ms │       51.10 ms │  1.05x slower │
│ QQuery 14    │  14.19 ms │       14.55 ms │     no change │
│ QQuery 15    │  25.55 ms │       25.85 ms │     no change │
│ QQuery 16    │  25.21 ms │       25.73 ms │     no change │
│ QQuery 17    │ 157.36 ms │      158.38 ms │     no change │
│ QQuery 18    │ 284.22 ms │      288.72 ms │     no change │
│ QQuery 19    │  37.24 ms │       37.34 ms │     no change │
│ QQuery 20    │  51.23 ms │       51.34 ms │     no change │
│ QQuery 21    │ 336.79 ms │      317.17 ms │ +1.06x faster │
│ QQuery 22    │  17.38 ms │       18.36 ms │  1.06x slower │
└──────────────┴───────────┴────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary             ┃           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD)             │ 1854.48ms │
│ Total Time (refactor_join2)   │ 1837.30ms │
│ Average Time (HEAD)           │   84.29ms │
│ Average Time (refactor_join2) │   83.51ms │
│ Queries Faster                │         5 │
│ Queries Slower                │         3 │
│ Queries with No Change        │        14 │
│ Queries with Failure          │         0 │
└───────────────────────────────┴───────────┘

@Dandandan
Copy link
Contributor Author

@alamb @martin-g
this is ready

Looks like tpc-h mem SF=1 also shows a small improvement for query 21, but smaller than SF=10

@Omega359
Copy link
Contributor

Omega359 commented Dec 3, 2025

@alamb let's fire some benchmarks? I mostly see improvement on SF=10 in memory on the standard set of benchmarks

I started some. It would be really nice to find some way for others to run their own -- I don't think it is great I am often the bottleneck.

Maybe I'll see if I can get an LLM to write me a script to do that...

You GH repo is public. I copied your scripts locally and slightly modified them for my env but I'm sure anyone should be able to do it.

@alamb
Copy link
Contributor

alamb commented Dec 3, 2025

@alamb let's fire some benchmarks? I mostly see improvement on SF=10 in memory on the standard set of benchmarks

I started some. It would be really nice to find some way for others to run their own -- I don't think it is great I am often the bottleneck.
Maybe I'll see if I can get an LLM to write me a script to do that...

You GH repo is public. I copied your scripts locally and slightly modified them for my env but I'm sure anyone should be able to do it.

Indeed!

I also indulged in some additional scripting to launch things from PRs -- #18115 (comment)

I'll maybe add some additional well known contributors too

@alamb
Copy link
Contributor

alamb commented Dec 3, 2025

show benchmark queue

@alamb
Copy link
Contributor

alamb commented Dec 3, 2025

🤖 Hi @alamb, you asked to view the benchmark queue (#18972 (comment)).

Job User Benchmarks Comment
18875.sh unknown tpch unknown
18766_3608566553.sh alamb default https://github.com/apache/datafusion/pull/18766#issuecomment-3608566553

@Dandandan
Copy link
Contributor Author

@alamb let's fire some benchmarks? I mostly see improvement on SF=10 in memory on the standard set of benchmarks

I started some. It would be really nice to find some way for others to run their own -- I don't think it is great I am often the bottleneck.
Maybe I'll see if I can get an LLM to write me a script to do that...

You GH repo is public. I copied your scripts locally and slightly modified them for my env but I'm sure anyone should be able to do it.

Yes the benchmarks can be run (and I usually do as part of PRs), but it is always nice to run on different environments to verify / compare. Nice to automate this stuff 🚀

@Dandandan
Copy link
Contributor Author

@milenkovicm perhaps you have time for review?

@milenkovicm
Copy link
Contributor

will try over weekend if not too late

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate optimizer Optimizer rules physical-plan Changes to the physical-plan crate sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Integrate BatchCoalescer into HashJoinExec and remove from CoalesceBatches optimization rule

5 participants