Skip to content

feat: Support Primitive Interface for Streaming and Chunked ZTDF Creation #2581

@jrschumacher

Description

@jrschumacher

Problem
The current Go SDK implementation does not support streaming or chunked I/O for TDF creation. This limitation poses significant challenges for applications that handle large datasets or require incremental data processing, such as multipart uploads to S3-compatible storage like S4, where the Content-Length of each part must be known before the upload. Holding an entire large TDF in memory is inefficient and often infeasible, limiting performance and scalability.

User Story
As a developer, I need a low-level API to create ZTDF files by writing encrypted data segments incrementally and in an arbitrary order. This will allow me to handle large files with low memory overhead and integrate with services that require chunked uploads. After all data segments are written, I want to call a finalization function that wraps the necessary keys, generates the manifest, and completes the TDF archive.

Misc
The proposed solution is to introduce a primitive writer interface. This interface would expose methods to initialize a TDF, write data segments, and finalize the TDF package.

A key concern is balancing this new flexibility with security to avoid creating a "footgun." Essential cryptographic and policy enforcement controls must remain non-configurable by the primitive interface to ensure consistency and security. If any such controls need to be exposed for advanced use cases, they should be explicitly marked and documented as internal-only to prevent unintended misuse by external developers.

Acceptance Criteria

Functional Requirements

  • A primitive writer interface is available for creating ZTDF files.
  • The interface allows for writing payload segments in any order (non-sequential).
  • The interface includes a finalization method that correctly writes the manifest, wraps keys, and closes the zip archive.
  • TDFs created using this interface are valid and can be successfully decrypted by standard TDF clients.
  • The final TDF size can be deterministically calculated before the TDF is fully written.

Non-Functional Requirements

  • Performance: The creation process should support streaming and not require the entire plaintext or ciphertext to be buffered in memory.
  • Memory: Memory consumption should be proportional to the size of a data chunk, not the entire file.
  • Security: The interface must not allow developers to bypass or weaken core cryptographic or policy enforcement mechanisms.
  • Documentation: Advanced or internal-only components of the interface should be clearly separated from the public-facing API and not included in general developer documentation.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions