Skip to content

implement GCS resumable uploads as a multipart upload method #504

@james-rms

Description

@james-rms

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

When performing a streaming upload to GCS, the only option in this crate is Multipart Uploads. This works the same way it does for S3, and has the same limitations: you cannot perform an upload with >10000 parts. This presents a challenge for large uploads with unknown size. You have to pick a part size large enough to allow the largest plausible upload size, but still small enough that your buffers don't grow too large, and your cost of retrying a part upload is not too great.

The Resumable upload API does not have this limitation, you can upload as many parts as you want at whatever size you want.

The downside of resumable uploads is that part uploads must be done serially, you cannot upload parts in parallel or out-of-order.

Describe the solution you'd like

I'd like the option to use resumable uploads when using put_multipart. I would configure this in the client builder.

Describe alternatives you've considered
I can't think of one.

Additional context

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions