feat: Introduce sst file format for btree global index #49

ChaomingZhangCN · 2026-01-07T05:36:40Z

Purpose

This PR only covers basic reading and writing of SST format files, data compression encoding and caching are not considered. Too much code in the PR will make it difficult to review, so I will continue work for this in the future.

DONE:

cache interface
memory slice & memory slice input & memory slice output
sst format file write & read
segment based bloomfilter & bitset

TODO:

block cache: lru?
compression: lz4, zstd, etc.(simd implementaion?)

Tests

./build/debug/paimon-common-sst-file-format-test
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from SstFileIOTest
[ RUN ] SstFileIOTest.TestSimple
[ OK ] SstFileIOTest.TestSimple (2 ms)
[----------] 1 test from SstFileIOTest (2 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (2 ms total)
[ PASSED ] 1 test.

./build/debug/paimon-common-test
[----------] 1 test from BitSetTest
[ RUN ] BitSetTest.TestBitSet
[ OK ] BitSetTest.TestBitSet (0 ms)
[----------] 1 test from BitSetTest (0 ms total)

[----------] 5 tests from BloomFilterTest
[ RUN ] BloomFilterTest.TestOneSegmentBuilder
[ OK ] BloomFilterTest.TestOneSegmentBuilder (0 ms)
[ RUN ] BloomFilterTest.TestEstimatedHashFunctions
[ OK ] BloomFilterTest.TestEstimatedHashFunctions (0 ms)
[ RUN ] BloomFilterTest.TestBloomNumBits
[ OK ] BloomFilterTest.TestBloomNumBits (0 ms)
[ RUN ] BloomFilterTest.TestBloomNumHashFunctions
[ OK ] BloomFilterTest.TestBloomNumHashFunctions (0 ms)
[ RUN ] BloomFilterTest.TestBloomFilter
[ OK ] BloomFilterTest.TestBloomFilter (2 ms)
[----------] 5 tests from BloomFilterTest (2 ms total)

lxy-9602 · 2026-01-08T12:34:00Z

src/paimon/common/io/cache/cache.cpp

@@ -0,0 +1,42 @@
+/*
+ * Copyright 2024-present Alibaba Inc.
+ *


Please update the copyright year from 2024 to 2026 for new files.

lxy-9602 · 2026-01-08T12:38:30Z

src/paimon/common/io/cache/cache.h

+class CacheValue {
+ public:
+    explicit CacheValue(std::shared_ptr<MemorySegment>& segment) : segment_(segment) {}
+


const std::shared_ptr& segment may be better.

lxy-9602 · 2026-01-08T12:40:13Z

src/paimon/common/io/cache/cache.h

+        std::function<std::shared_ptr<CacheValue>(const std::shared_ptr<CacheKey>&)> supplier) = 0;
+
+    virtual void Put(const std::shared_ptr<CacheKey>& key, std::shared_ptr<CacheValue>& value) = 0;
+


const std::shared_ptr& value may be better.

lxy-9602 · 2026-01-08T12:44:28Z

src/paimon/common/io/cache/cache_key.cpp

+    size_t seed = 0;
+    seed ^=
+        std::hash<std::string>{}(file_path_) + 0x9e3779b97f4a7c15ULL + (seed << 6) + (seed >> 2);
+    seed ^= std::hash<int64_t>{}(position_) + 0x9e3779b97f4a7c15ULL + (seed << 6) + (seed >> 2);


Consider converting 0x9e3779b97f4a7c15ULL to a static constant

lxy-9602 · 2026-01-08T12:47:47Z

src/paimon/common/io/cache/cache_manager.cpp

+    std::function<Result<MemorySegment>(const std::shared_ptr<CacheKey>&)> reader) {
+    auto& cache = key->IsIndex() ? index_cache_ : data_cache_;
+    auto supplier = [&reader](const std::shared_ptr<CacheKey>& k) -> std::shared_ptr<CacheValue> {
+        auto segment = reader(k).value();


Capturing &reader by reference may risk memory leaks. May consider capturing by value instead. Also ensure Result is .ok() before calling .value() to avoid undefined behavior.

lxy-9602 · 2026-01-08T13:47:57Z

src/paimon/common/sst/block_aligned_type.h

+        return BlockAlignedType::UNALIGNED;
+    } else {
+        throw std::invalid_argument("Invalid block aligned type: " + std::to_string(v));
+    }


Also, may return Result<BlockAlignedType>

lxy-9602 · 2026-01-08T13:49:42Z

src/paimon/common/sst/block_cache.h

+    BlockCache(std::string& file_path, std::shared_ptr<InputStream>& in,
+               std::shared_ptr<MemoryPool>& pool, std::unique_ptr<CacheManager>& cache_manager)
+        : file_path_(file_path), in_(in), pool_(pool), cache_manager_(std::move(cache_manager)) {}
+


may pass const shared_ptr& or unique_ptr&&

lxy-9602 · 2026-01-08T14:02:48Z

src/paimon/common/sst/block_entry.h

+ public:
+    BlockEntry(std::shared_ptr<MemorySlice>& key, std::shared_ptr<MemorySlice>& value)
+        : key_(key), value_(value) {}
+


Consider making BlockEntry a struct if it only has Get methods.

lxy-9602 · 2026-01-09T05:46:11Z

src/paimon/common/sst/block_handle.h

+
+ public:
+    static constexpr int32_t MAX_ENCODED_LENGTH = 9 + 5;
+


Can you add a comment explaining the value of MAX_ENCODED_LENGTH?

lxy-9602 · 2026-01-09T05:48:21Z

src/paimon/common/sst/block_handle.cpp

+    auto output = std::make_shared<MemorySliceOutput>(MAX_ENCODED_LENGTH, pool);
+    output->WriteVarLenLong(offset_);
+    output->WriteVarLenLong(size_);
+    return output->ToSlice();


I'm not sure if reading uses an int followed by a long (ReadVarLenLong for offset and ReadVarLenInt for size), while writing uses two longs. Is this correct? Consider adding a test to verify.

lxy-9602 · 2026-01-09T05:58:20Z

src/paimon/common/sst/block_iterator.cpp

+std::unique_ptr<BlockEntry> BlockIterator::Next() {
+    if (!HasNext()) {
+        throw std::invalid_argument("no such element");
+    }


May use bad status instead of exceptions.

lxy-9602 · 2026-01-09T06:07:18Z

src/paimon/common/sst/block_iterator.cpp

+    }
+
+    return false;
+}


Please add a test for BlockReader and BlockIterator.

lxy-9602 · 2026-01-09T06:09:37Z

src/paimon/common/sst/block_reader.h

+class BlockIterator;
+
+/** Reader for a block. */
+class BlockReader : public std::enable_shared_from_this<BlockReader> {


Consider using /// for class or interface comments to maintain a consistent format.

lxy-9602 · 2026-01-09T06:20:36Z

src/paimon/common/sst/block_reader.cpp

+        auto data = block->Slice(0, index_offset);
+        auto index = block->Slice(index_offset, index_length);
+        return std::make_shared<UnAlignedBlockReader>(data, index, comparator);
+    }


Can 5 be a static constant rather than a magic number? Or add comments for it.

lxy-9602 · 2026-01-09T06:23:57Z

src/paimon/common/sst/block_trailer.cpp

+std::string BlockTrailer::ToString() const {
+    return "BlockTrailer{compression_type=" + std::to_string(compression_type_) + ", crc32c_=0x" +
+           std::to_string(crc32c_) + "}";
+}


May add a conversion to ensure the return value is a hexadecimal string for crc32c.

lxy-9602 · 2026-01-09T06:29:51Z

src/paimon/common/sst/block_writer.cpp

+    if (aligned_) {
+        int currentSize = endPosition - startPosition;
+        if (aligned_size_ == 0) {
+            aligned_size_ = currentSize;


current_size & end_position & start_position

lxy-9602 · 2026-01-09T06:50:08Z

src/paimon/common/sst/sst_file_writer.h

+    BlockWriter* IndexWriter() const {
+        return index_block_writer_.get();
+    }
+


Test interface can be private.

lxy-9602 · 2026-01-09T06:51:27Z

src/paimon/common/sst/sst_file_writer.h

+    std::shared_ptr<Bytes> last_key_;
+
+    const std::shared_ptr<MemoryPool>& pool_;
+


Ensure that obtaining a shared_ptr reference does not introduce any risk of memory leaks？

lxy-9602 · 2026-01-09T06:54:04Z

src/paimon/common/sst/sst_file_writer.cpp

+    if (!handle.ok()) {
+        return handle.status();
+    }
+


May use PAIMON_ASSIGN_OR_RAISE() (ps: Ensure that the macro PAIMON_ASSIGN_OR_RAISE always includes the return type not auto ).

lxy-9602 · 2026-01-09T07:09:12Z

src/paimon/common/utils/bloom_filter.h

+
+ private:
+    static constexpr int32_t BYTE_SIZE = 8;
+


Can the existing BloomFilter64 and BitSet classes in paimon/common/utils/ meet your requirements for bloom filter file index?

lxy-9602 · 2026-01-09T07:10:40Z

src/paimon/common/io/cache/cache_key.h

+
+    bool IsIndex() override;
+
+    int64_t Position();


Get method can be const.

ChaomingZhangCN added 3 commits January 5, 2026 15:36

feat: introduce sst file format for btree global index

f01566f

feat: introduce sst file format for btree global index

3a8f42d

fix bloom filter

96ec401

ChaomingZhangCN changed the title ~~feat: introduce sst file format for btree global index~~ feat: sst file format for btree global index Jan 7, 2026

ChaomingZhangCN added 2 commits January 7, 2026 13:54

minor fix

69e1434

fix compile

95e2012

ChaomingZhangCN changed the title ~~feat: sst file format for btree global index~~ feat: Introduce sst file format for btree global index Jan 7, 2026

ChaomingZhangCN and others added 13 commits January 7, 2026 16:55

minor fix

e0c876e

fix compile

73bc0c2

Merge branch 'main' into sst-file-format

0fdc643

add virtual

3ffcf50

address

e08c162

Merge branch 'main' into sst-file-format

592df79

address

573f843

address

a5c8c53

fix tests

a56c96c

fix tests

757d212

fix tests

a0cb2a7

fix tests

bfa93f4

fix tests

dfe0527

lxy-9602 reviewed Jan 9, 2026

View reviewed changes

		std::function<std::shared_ptr<CacheValue>(const std::shared_ptr<CacheKey>&)> supplier) = 0;

		virtual void Put(const std::shared_ptr<CacheKey>& key, std::shared_ptr<CacheValue>& value) = 0;

		std::shared_ptr<Bytes> last_key_;

		const std::shared_ptr<MemoryPool>& pool_;

feat: Introduce sst file format for btree global index #49

Are you sure you want to change the base?

feat: Introduce sst file format for btree global index #49

Uh oh!

Conversation

ChaomingZhangCN commented Jan 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

DONE:

TODO:

Tests

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lxy-9602 Jan 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ChaomingZhangCN commented Jan 7, 2026 •

edited

Loading

lxy-9602 Jan 9, 2026 •

edited

Loading