AWSS3TransferUtility errors on partial file creation for larger uploads

[creitz]

Describe the bug
When performing a multipart upload, the AWSS3TransferUtility often fails with

Error while creating temporary file for partial file: ...
Unable to process part # ...

To Reproduce
Incurred from calling AWSS3TransferUtility.uploadFileUsingMultiPart with a large file, > 5 GB.
Reduced storage on the device probably makes this more likely to occur as well.

Observed Behavior
When performing a multipart upload, the AWSS3TransferUtility creates a temporary file for each subtask. However, it does this even for subtasks that it is not starting because they are beyond the multipart concurrency limit.

This happens here:

aws-sdk-ios/AWSS3/AWSS3TransferUtility.m

Line 1248 in d3c1285

subTaskCreationError = [self createUploadSubTask:transferUtilityMultiPartUploadTask subTask:subTask startTransfer:NO internalDictionaryToAddSubTaskTo:transferUtilityMultiPartUploadTask.waitingPartsDictionary];

For relatively small files, this is okay. But for larger files (perhaps 5+ GB) this often produces the errors listed above.

I suspect this is because partial files are created for the entire upload up front when all of the subtasks are being created, even though only a few of the subtasks (up to multiPartConcurrencyLimit) are actually going to be started.

Expected Behavior
It seems to me like it would be much more space efficient to only create the temporary file part for the subtask once the subtask actually starts. This would save a lot of processing time up front, since hundred or even thousands of parts need to be processed in the beginning; as well as space while uploading, since only the parts that are uploading would be duplicated in storage.

I don't pretend to be well-versed in all the details of the SDK, so perhaps there is a reason this isn't doable - I see that for generating presigned url requests, if useContentMD5 is enabled, then the contents of the partial file are needed to generate that; however, if that setting is false (NO), then perhaps the partial file creation can be postponed to when needed.

Stack Trace
N/A

Code Snippet
Please provide a snippet of the code causing the issue or how you are using the service that has the bug.

Incurred from calling AWSS3TransferUtility.uploadFileUsingMultiPart with a large file.

Unique Configuration
N/A

Areas of the SDK you are using (AWSMobileClient, Cognito, Pinpoint, IoT, etc)?
AWSS3

Screenshots
N/A

Environment(please complete the following information):

• SDK Version: 5.38.1
• Dependency Manager: Cocoapods
• Swift Version : 5.9
• Xcode Version: 16.3

Device Information (please complete the following information):

• Device: iPhone 16 Pro
• iOS Version: 18.4.1

Additional context
N/A

Relevant Console Output
Error while creating temporary file for partial file: ...
Unable to process part # ...

[harsh62]

@creitz Thanks for raising the issue. I am tagging this as a bug, and one of our engineers will investigate this further.

[operezdev]

I am experiencing the same issue described in this report.

com.amazonaws.AWSS3TransferUtilityErrorDomain - 2: Error Domain=com.amazonaws.AWSS3TransferUtilityErrorDomain Code=2 "(null)" UserInfo={Message=Unable to process Part #: 2}

Let me know if you need more information.