Skip to content

Conversation

@ShreyasSinha
Copy link
Collaborator

@ShreyasSinha ShreyasSinha commented Dec 22, 2025

Implementing a WriteChannel for uploading objects to Google Cloud Storage using the multipart upload API.
We breaks the data into 5MB chunks and uploads them in parallel using a fixed-size thread pool.
When the channel is closed, it sends a request to GCS to complete the multipart upload, assembling the parts into the final object.

Sample Benchmark: (Writing 5GB via cloudtop)
Using JSON writer with XML MPU API (Pool_SIZE = 16)
--- Aggregate Metrics ---
Total time: 21.179 s
File size: 5000.000 MB
Throughput: 236.082 MB/s

Using gRPC (existing)
--- Aggregate Metrics ---
Total time: 52.980 s
File size: 5000.000 MB
Throughput: 94.376 MB/s

Code used for benchmarking: https://github.com/ShreyasSinha/gcs-xml-mpu-benchmark/blob/main/src/main/java/org/example/ResumableUploadBenchmark.java

Test failures expected as its a prototype code.

@product-auto-label product-auto-label bot added size: m Pull request size is medium. api: storage Issues related to the googleapis/java-storage API. labels Dec 22, 2025
@ShreyasSinha ShreyasSinha changed the title Sample code to prototype using MPU to implement write methods that currently use resumable uploads chore: Sample code to prototype using MPU to implement write methods that currently use resumable uploads Dec 22, 2025
@product-auto-label product-auto-label bot added size: l Pull request size is large. and removed size: m Pull request size is medium. labels Dec 22, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api: storage Issues related to the googleapis/java-storage API. size: l Pull request size is large.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant