Skip to content

Conversation

@aryan-cloud
Copy link

Rationale for this change

Fuse definition level counting into a bitmap-based pass to reduce extra memory traversal and improve cache/SIMD efficiency, addressing the TODO in column_reader.cc.

What changes are included in this PR?

Replace std::count with a bitmap/PopCount helper for counting max definition levels in ReadLevels.

Are these changes tested?

yes
cpp/build-asan/debug/parquet-encoding-test --gtest_filter=*
cpp/build-asan/debug/parquet-arrow-internals-test --gtest_filter=*

Are there any user-facing changes?

No.

@aryan-cloud aryan-cloud requested a review from wgtmac as a code owner January 6, 2026 03:10
@github-actions
Copy link

github-actions bot commented Jan 6, 2026

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

See also:

Comment on lines +103 to +113
int64_t offset = 0;
for (; offset + 64 <= num_levels; offset += 64) {
const uint64_t bitmap =
internal::GreaterThanBitmap(levels + offset, 64, rhs);
count += static_cast<int64_t>(bit_util::PopCount(bitmap));
}
if (offset < num_levels) {
const uint64_t bitmap =
internal::GreaterThanBitmap(levels + offset, num_levels - offset, rhs);
count += static_cast<int64_t>(bit_util::PopCount(bitmap));
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
int64_t offset = 0;
for (; offset + 64 <= num_levels; offset += 64) {
const uint64_t bitmap =
internal::GreaterThanBitmap(levels + offset, 64, rhs);
count += static_cast<int64_t>(bit_util::PopCount(bitmap));
}
if (offset < num_levels) {
const uint64_t bitmap =
internal::GreaterThanBitmap(levels + offset, num_levels - offset, rhs);
count += static_cast<int64_t>(bit_util::PopCount(bitmap));
}
for (int64_t offset = 0; offset < num_levels; offset += 64) {
const int64_t chunk_size = std::min<int64_t>(64, num_levels - offset);
const uint64_t bitmap =
internal::GreaterThanBitmap(levels + offset, static_cast<int>(chunk_size), rhs);
count += static_cast<int64_t>(bit_util::PopCount(bitmap));
}

I suggest simplifying the two loops into one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants