Indeterministic (out-of-order) streaming support in Go SDK

[jrschumacher]

Indeterministic (out-of-order) Streaming Support in Go SDK

There are cases where streaming is indeterministic—for example, when the total file size or part sizes are not known ahead of time—and we need the ability to encrypt data in chunks and then piece it back together with a manifest after the last chunk is complete.

With the AWS S3 CLI, the user has the ability to specify the multipart chunk size, but this requires advanced knowledge of the SDK’s encryption process to compute the size appropriately. Additionally, all data would need to be stored in plaintext or encrypted with a symmetric key until all parts are available, then decrypted to re-encrypt as part of the TDF process.

Goal:
Enable the Go SDK to support storing parts as encrypted chunks, which can be assembled as the final segments of the TDF. Once the last chunk is encrypted, the parts would be joined, the manifest written, and a valid TDF created—without ever needing to keep the full plaintext or re-encrypt previously uploaded data.

---

User Story

As a developer using the Go SDK for TDF, I want to upload large files in encrypted chunks of unknown size so I can assemble them into a valid TDF at the end, without storing plaintext or re-encrypting intermediate data.

---

Constraints & Assumptions

• The solution should be compatible with existing TDF manifests and chunking logic.
• Chunks must be encrypted immediately—no intermediate plaintext storage.
• The final TDF output must be valid and compatible with standard TDF readers.
• Security and integrity of each chunk and the assembled file must be maintained.

---

Acceptance Criteria

• The Go SDK allows encryption and upload of arbitrary-sized chunks without needing to know the total size in advance.
• The SDK supports assembling the encrypted chunks into a valid TDF file, with the manifest written after the last chunk.
• The final TDF is compatible with standard TDF readers and passes integrity/decryption tests.
• Encrypted chunks are never stored or transmitted as plaintext at any point.
• Documentation is updated to describe usage patterns for indeterministic streaming.
• Integration tests cover:
  • Upload of a large file in random-sized chunks.
  • Correct manifest generation after the last chunk.
  • Decryption and integrity verification of the final TDF.

---

[dmihalcik-virtru]

I think the need may go a bit beyond 'indeterminate' streaming - we also want to support disconnected or distributed middleware type writers. e.g. we want to be able to write segmenents x and y concurrently, possibly on different pods. So we need to consider how we can achieve:

• crash protection: writes should be able to restart of the SDK closes and restarts
• segment independence: writing separate segments should be able to occur without knowledge of each other. Sufficient knowledge includes:
  • order: segment X occurs before segment Y
  • non-overlap: segment X and Y do not overlap
• Shared initial knowledge: there is some shared knowledge at the initialization phase. This should be kept small and preferrably near contstant size
• 'Finishing' action: after all segments are processed, they produce some output and metadata. This can be correlated into one or more 'finish' segments that are placed before/after the segments (ie the zip trailer, additional envelope bits, and manifest)

The use case here is for a front-end to a distributed file system, so we can't expect writes to be happening on the same server, IIUC. However, there must be sufficient information communicated to achieve what we want

[wwchung11]

Sorry, if this is a dumb question. I know the file size is indeterminate on the receiver side
but are you saying the file/tdf may be indeterminate sized on the sender side?
Like client is sending in data via "|" eg $ cat someLargeFile.tdf | app-go_sdk.exe

[jrschumacher]

@wwchung11 In some situations, you might be proxying and protecting data between existing (non-OpenTDF aware) clients and a remote storage service. In this case, as a developer, I wouldn’t be able to synchronize the chunk size or order of the client’s upload behavior with the expectations of the OpenTDF proxy.

For example, some cloud storage SDKs manage multipart uploads in non-deterministic ways—varying chunk sizes and retries independently. If I were to intercept this traffic, I couldn’t guarantee alignment between the client’s transfer pattern and OpenTDF’s wrapping requirements.

[noahdav]

For our use case we are trying to read and write chunks of the file. We provide the user with a decrypted view of the file while maintain the protected TDF file in the Windows kernel. When writes occur we know the offset and length, but this is usually just a chunk of the file aligned to the size of a page in memory.

We would need the ability to write chunks of the file without the entire file needing to be in memory. One option is to encrypt into intermediate temp files, then play them back to the streaming service to create the final file. This would allow us to guarantee that we have all the chunks before we write the final file. We could also just stream them directly without the temp files if the streaming service can handle this.

For decrypt we would also need the ability able to open the TDF and ask for only chunks at a time without reopening the TDF file.