Skip to content

Expose set_pledged_src_size as a public API for zstd compression #329

@shrekris-openai

Description

@shrekris-openai

Issue

async-compression offers no way to call set_pledged_src_size() on the zstd encoder. Could we provide a public API in async-compression to call this?

Context

zstd encoding can include an optional Frame_Content_Size value. Decompression libraries such as Python's zstandard provide APIs (e.g. decompress) that use this value to decompress the data in a single operation. Without it, these libraries must treat the encoded bytes as a stream.

Writing Frame_Content_Size requires two steps (from the zstd manual):

  1. Enable ZSTD_c_contentSizeFlag. async-compression already offers a CParameter method to do this.
  2. Call ZSTD_CCtx_setPledgedSrcSize() to set the decompressed data size. The zstd crate already exposes the public method set_pledged_src_size() for this, but it's not exposed by the async-compression crate.

When set_pledged_src_size is not called, the encoded bytes won't have a Frame_Content_Size value.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions