chore: Sample code to prototype using MPU to implement write methods that currently use resumable uploads #3448
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Implementing a WriteChannel for uploading objects to Google Cloud Storage using the multipart upload API.
We breaks the data into 5MB chunks and uploads them in parallel using a fixed-size thread pool.
When the channel is closed, it sends a request to GCS to complete the multipart upload, assembling the parts into the final object.
Sample Benchmark: (Writing 5GB via cloudtop)
Using JSON writer with XML MPU API (Pool_SIZE = 16)
--- Aggregate Metrics ---
Total time: 21.179 s
File size: 5000.000 MB
Throughput: 236.082 MB/s
Using gRPC (existing)
--- Aggregate Metrics ---
Total time: 52.980 s
File size: 5000.000 MB
Throughput: 94.376 MB/s
Code used for benchmarking: https://github.com/ShreyasSinha/gcs-xml-mpu-benchmark/blob/main/src/main/java/org/example/ResumableUploadBenchmark.java
Test failures expected as its a prototype code.