Add Spark accelerators

[Iskander14yo]

Motivation

There are several Spark plugins and extensions that claim to offer significantly better performance than vanilla Spark for analytical workloads -for example, Apache Comet (based on DataFusion), Blaze (DataFusion), and Apache Gluten (mainly Velox, ClickHouse). Since ClickBench already allows GPU-based engines (#410), NVIDIA RAPIDS could also be considered.

I believe these are worth evaluating for a few reasons:

• Independent benchmark: These accelerators are typically benchmarked only on their own TPC-H/TPC-DS variants. Including them in ClickBench would offer a more independent comparison. This is valuable to both users and developers.
• Distributed vs. single-node overhead: Some of these tools have single-node counterparts (e.g., DataFusion). Measuring how much overhead comes from different factors (such as Spark’s architecture, quality of implementation) is useful in itself.
• Spark user insights: For Spark users, this shows how much they can gain (or lose) by adopting such plugins (which are mostly easy to install having already deployed Spark) to compare with other distributed engines and Spark itself.

Request

Would you be open to including these in ClickBench? If I checked correctly, no similar issue has been created yet.
If yes, I’d be glad to take on this task and submit the necessary PRs incrementally (likely one per engine).

[rschu1ze]

Would you be open to including these in ClickBench?

Yes, that would be nice. Please send PR(s).

[Iskander14yo]

Can I ask some questions?

Do we pursue a goal to put these engines in the most equal conditions, or is that not necessary? I mean mainly Spark versions here — since hardware is fixed, and I haven’t identified other major factors that could impact results yet.

ClickBench isn’t Spark-specific, so it seems fine to use whatever works best per engine. But if we want comparisons to be more fair (and helpful for Spark users), maybe it makes sense to align Spark versions across vanilla Spark, Comet, Gluten, etc.

So first option is to keep Spark version consistent within the Spark “family,” and allow exceptions with clear notes.
Another option is to use "latest stable Spark version per engine" approach. My thoughts were:

• Current Spark version in repo is 4.0 and now it is supported only by Comet and only for testing;
• Version alignment might be hard to maintain (e.g., today Gluten-on-Clikchouse only supports Spark 3.2/3.3);
• Patch versions also differ often, though it doesn't seem critical.

Let me know which direction seems reasonable for you. Currently I'm leaning towards second option because it's easier to maintain and still "fair" (we use same VM anyway and let each engine use latest Spark version it supports).

[SparkApplicationMaster]

@Iskander14yo +1 for the second "full power" approach, outdated frameworks will become only more outdated...

If the engines support pyspark, you can start with some of this code:
https://github.com/ClickHouse/ClickBench/blob/main/spark/query.py
https://github.com/ClickHouse/ClickBench/blob/main/sail/query.py
For example, sail is a spark replacement framework written in pure rust that uses spark connect and datafusion, so no code changes were needed except the SparkSession configuration, at least on ClickBench

It would be nice to see if the engines you mentioned can do ClickBench without rewriting the code or queries
Maybe some common module can be created with common functions for all the engines?

[Iskander14yo]

Thanks for the feedback!

Yep, I definitely want to reuse current Spark code as much as possible. At least because part of these backends claims to work "without code changes". In practice, that's not always true, so we might face some difficulties here.

I'm not sure about introducing a shared module though. It could become harder to maintain over time, especially as different engines diverge and require different setup or tweaks. And ClickBench doesn’t seem to be structured for cross-engine code reuse at the moment.

That said, I’ll definitely document any reusable patterns and relations in the README. Moreover, I’ll start sending PRs soon — and from there we can see whether some form of reuse makes sense in practice.

[rschu1ze]

And ClickBench doesn’t seem to be structured for cross-engine code reuse at the moment.

True, though this was not an intentional decision, and it is fine to share code (e.g. setup code) to cover the Spark situation.

I don't have a strong opinion on the first vs. the second approach. Once GPUs enter the game, a "fair" comparison with CPU-only databases is no longer feasible anyways. But if at last all Spark submissions want to standardize on the same hardware platform (with a GPU), that would be helpful to compare performance across different Spark flavors.

[Iskander14yo]

I have another Q:
In general, these engines don’t fully support the same set of operators, functions, or expressions as Spark. In such cases, they (often) gracefully fall back to Spark execution or (rarely) fail due to semantic mismatches between engines.

Currently I observe falling back while failing apparently is not the case here but we may take it into account as well. Should we rewrite queries in those cases, or keep them as-is?

Reasons to rewrite:

• Better performance potential and more interesting comparison since accelerators do full query execution, not partial.
• I consider it fair realistic because users are generally OK with minor query changes if it helps to deal with heavy workloads.
• Bench queries aren’t identical across engines anyway. Some adaptation is expected, so we can consider rewriting as just another backend-specific adjustment (though not that serious to consider it tuned).
• I don't think this fits the case of "query optimization" since queries are mostly simple (and I expect them to be more complex afterwards).

Reasons not to rewrite:

• There is no guarantee that after rewriting query engine will perform better. Queries tend to become more complex, optimizer may fail and so on.
• If an engine falls back to Spark, it doesn’t mean the system failed — it’s still a legitimate execution path. (That’s likely why devs usually allocate memory for Spark as well, to ensure fallback queries don’t get stuck or crushed.) So there is no necessity in rewriting in that case.
• Tracking (un-)supported things may become burden in the future. Each version update should also include queries review as I see it to come back to original ones where possible. It doesn't seem difficult though.

In fact, if correctness of this decision is concern we can avoid dichotomy and set 2 launches - with both default and rewritten queries (same folder, different queries.sql and results). That's the option I prefer now. WDYT?