put_object upload speed regression

[tristan-morris]

Describe the bug

put_object would previously reach my line rate and stay there, whereas at least on latest/1.100.0 will have a small peak at the start, and then significantly drop off until it eventually stalls. The same 'slowness' occurs with multipart and smaller files.

I'm attempting to upload an 800mb file, same issue with smaller.

This chart below shows the throughput for both versions as listed below.

Reaches ~28mbps and stays there.

aws-config = { version = "=1.0.0", features = ["behavior-version-latest"] }
aws-sdk-s3 = "=1.32.0"

vs

Reaches ~2mbps and drops off, usually backing off to zero and stalling.

aws-config = { version = "1.8.3", features = ["behavior-version-latest"] }
aws-sdk-s3 = "1.100.0"

I only found this after our monitoring system identified a heap of s3 upload failures.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

The s3 upload would occur at full speed.

Current Behavior

Reaches ~2mbps and drops off, usually backing off to zero and stalling.

Reproduction Steps

Cargo.toml

# aws-config = { version = "1.8.3", features = ["behavior-version-latest"] }
# aws-sdk-s3 = "1.100.0"
aws-config = { version = "=1.0.0", features = ["behavior-version-latest"] }
aws-sdk-s3 = "=1.32.0"

Custom S3 provider in this case is cloudflare r2, but I assume it'll also impact other providers.

    let region_provider = Region::new(config.region_name.clone());

    let credentials = Credentials::new(
        config.access_key.clone(),
        config.secret_key.clone(),
        None,
        None,
        "api-custom-s3-provider",
    );

    let sdk_config = aws_config::from_env()
        .region(region_provider)
        .endpoint_url(config.url.clone())
        .credentials_provider(credentials)
        .retry_config(RetryConfig::standard().with_max_attempts(2))
        .load()
        .await;

    let client = Client::new(&sdk_config);

let stream = ByteStream::from_path(local_path).await?;

client
            .put_object()
            .bucket(self.s3_config.bucket.clone())
            .key(key)
            .body(stream)
            .send()
            .await?

Possible Solution

Unsure, I'll likely look into this deeper at some point. Possibly related to hyper and feels very TCP window-y related.

Additional Information/Context

No response

Version

cargo tree | grep aws-
    Blocking waiting for file lock on package cache
    Updating crates.io index
    Blocking waiting for file lock on package cache
    Blocking waiting for file lock on package cache
│   │   │   │   ├── aws-lc-rs v1.13.3
│   │   │   │   │   ├── aws-lc-sys v0.30.0
│   │   │   │   │   ├── aws-lc-rs v1.13.3 (*)
│   ├── aws-config v1.8.3
│   │   ├── aws-credential-types v1.2.4
│   │   │   ├── aws-smithy-async v1.2.5
│   │   │   ├── aws-smithy-runtime-api v1.8.5
│   │   │   │   ├── aws-smithy-async v1.2.5 (*)
│   │   │   │   ├── aws-smithy-types v1.3.2
│   │   │   ├── aws-smithy-types v1.3.2 (*)
│   │   ├── aws-runtime v1.5.9
│   │   │   ├── aws-credential-types v1.2.4 (*)
│   │   │   ├── aws-sigv4 v1.3.3
│   │   │   │   ├── aws-credential-types v1.2.4 (*)
│   │   │   │   ├── aws-smithy-eventstream v0.60.10
│   │   │   │   │   ├── aws-smithy-types v1.3.2 (*)
│   │   │   │   ├── aws-smithy-http v0.62.2
│   │   │   │   │   ├── aws-smithy-eventstream v0.60.10 (*)
│   │   │   │   │   ├── aws-smithy-runtime-api v1.8.5 (*)
│   │   │   │   │   ├── aws-smithy-types v1.3.2 (*)
│   │   │   │   ├── aws-smithy-runtime-api v1.8.5 (*)
│   │   │   │   ├── aws-smithy-types v1.3.2 (*)
│   │   │   ├── aws-smithy-async v1.2.5 (*)
│   │   │   ├── aws-smithy-eventstream v0.60.10 (*)
│   │   │   ├── aws-smithy-http v0.62.2 (*)
│   │   │   ├── aws-smithy-runtime v1.8.5
│   │   │   │   ├── aws-smithy-async v1.2.5 (*)
│   │   │   │   ├── aws-smithy-http v0.62.2 (*)
│   │   │   │   ├── aws-smithy-http-client v1.0.6
│   │   │   │   │   ├── aws-smithy-async v1.2.5 (*)
│   │   │   │   │   ├── aws-smithy-runtime-api v1.8.5 (*)
│   │   │   │   │   ├── aws-smithy-types v1.3.2 (*)
│   │   │   │   ├── aws-smithy-observability v0.1.3
│   │   │   │   │   └── aws-smithy-runtime-api v1.8.5 (*)
│   │   │   │   ├── aws-smithy-runtime-api v1.8.5 (*)
│   │   │   │   ├── aws-smithy-types v1.3.2 (*)
│   │   │   ├── aws-smithy-runtime-api v1.8.5 (*)
│   │   │   ├── aws-smithy-types v1.3.2 (*)
│   │   │   ├── aws-types v1.3.8
│   │   │   │   ├── aws-credential-types v1.2.4 (*)
│   │   │   │   ├── aws-smithy-async v1.2.5 (*)
│   │   │   │   ├── aws-smithy-runtime-api v1.8.5 (*)
│   │   │   │   ├── aws-smithy-types v1.3.2 (*)
│   │   ├── aws-sdk-sso v1.78.0
│   │   │   ├── aws-credential-types v1.2.4 (*)
│   │   │   ├── aws-runtime v1.5.9 (*)
│   │   │   ├── aws-smithy-async v1.2.5 (*)
│   │   │   ├── aws-smithy-http v0.62.2 (*)
│   │   │   ├── aws-smithy-json v0.61.4
│   │   │   │   └── aws-smithy-types v1.3.2 (*)
│   │   │   ├── aws-smithy-runtime v1.8.5 (*)
│   │   │   ├── aws-smithy-runtime-api v1.8.5 (*)
│   │   │   ├── aws-smithy-types v1.3.2 (*)
│   │   │   ├── aws-types v1.3.8 (*)
│   │   ├── aws-sdk-ssooidc v1.79.0
│   │   │   ├── aws-credential-types v1.2.4 (*)
│   │   │   ├── aws-runtime v1.5.9 (*)
│   │   │   ├── aws-smithy-async v1.2.5 (*)
│   │   │   ├── aws-smithy-http v0.62.2 (*)
│   │   │   ├── aws-smithy-json v0.61.4 (*)
│   │   │   ├── aws-smithy-runtime v1.8.5 (*)
│   │   │   ├── aws-smithy-runtime-api v1.8.5 (*)
│   │   │   ├── aws-smithy-types v1.3.2 (*)
│   │   │   ├── aws-types v1.3.8 (*)
│   │   ├── aws-sdk-sts v1.80.0
│   │   │   ├── aws-credential-types v1.2.4 (*)
│   │   │   ├── aws-runtime v1.5.9 (*)
│   │   │   ├── aws-smithy-async v1.2.5 (*)
│   │   │   ├── aws-smithy-http v0.62.2 (*)
│   │   │   ├── aws-smithy-json v0.61.4 (*)
│   │   │   ├── aws-smithy-query v0.60.7
│   │   │   │   ├── aws-smithy-types v1.3.2 (*)
│   │   │   ├── aws-smithy-runtime v1.8.5 (*)
│   │   │   ├── aws-smithy-runtime-api v1.8.5 (*)
│   │   │   ├── aws-smithy-types v1.3.2 (*)
│   │   │   ├── aws-smithy-xml v0.60.10
│   │   │   ├── aws-types v1.3.8 (*)
│   │   ├── aws-smithy-async v1.2.5 (*)
│   │   ├── aws-smithy-http v0.62.2 (*)
│   │   ├── aws-smithy-json v0.61.4 (*)
│   │   ├── aws-smithy-runtime v1.8.5 (*)
│   │   ├── aws-smithy-runtime-api v1.8.5 (*)
│   │   ├── aws-smithy-types v1.3.2 (*)
│   │   ├── aws-types v1.3.8 (*)
│   ├── aws-sdk-s3 v1.100.0
│   │   ├── aws-credential-types v1.2.4 (*)
│   │   ├── aws-runtime v1.5.9 (*)
│   │   ├── aws-sigv4 v1.3.3 (*)
│   │   ├── aws-smithy-async v1.2.5 (*)
│   │   ├── aws-smithy-checksums v0.63.5
│   │   │   ├── aws-smithy-http v0.62.2 (*)
│   │   │   ├── aws-smithy-types v1.3.2 (*)
│   │   ├── aws-smithy-eventstream v0.60.10 (*)
│   │   ├── aws-smithy-http v0.62.2 (*)
│   │   ├── aws-smithy-json v0.61.4 (*)
│   │   ├── aws-smithy-runtime v1.8.5 (*)
│   │   ├── aws-smithy-runtime-api v1.8.5 (*)
│   │   ├── aws-smithy-types v1.3.2 (*)
│   │   ├── aws-smithy-xml v0.60.10 (*)
│   │   ├── aws-types v1.3.8 (*)
│   │   │   │   ├── aws-lc-rs v1.13.3 (*)
│   │   │   │   │   ├── aws-lc-rs v1.13.3 (*)
│   │   │   ├── aws-config v1.8.3 (*)
│   │   │   ├── aws-sdk-sqs v1.78.0
│   │   │   │   ├── aws-credential-types v1.2.4 (*)
│   │   │   │   ├── aws-runtime v1.5.9 (*)
│   │   │   │   ├── aws-smithy-async v1.2.5 (*)
│   │   │   │   ├── aws-smithy-http v0.62.2 (*)
│   │   │   │   ├── aws-smithy-json v0.61.4 (*)
│   │   │   │   ├── aws-smithy-runtime v1.8.5 (*)
│   │   │   │   ├── aws-smithy-runtime-api v1.8.5 (*)
│   │   │   │   ├── aws-smithy-types v1.3.2 (*)
│   │   │   │   ├── aws-types v1.3.8 (*)

Environment details (OS name and version, etc.)

Ubuntu 24.04.2 LTS on WSL (dev box), also in Debian LTS Bookworm (prod containers)

Logs

No response

[ysaito1001]

Thank you for reporting this.

I'll likely look into this deeper at some point

That would be greatly appreciated. Since there have been many changes between versions 1.32.0 and 1.100.0 of aws-sdk-s3, pinpointing the area of interest will help us assist you better.

In addition, enabling tracing-subscriber with trace-level logging might reveal differences in behavior that could help narrow things down.

[landonxjames]

The biggest change in that time period would likely have been with 1.79.0 (along with aws-smithy-runtime 1.8.0). With that version we changed the default HTTP client from Hyper 0x to Hyper 1x. If you were going to bisect the changes to find where this started that would be a good place to start.

[tristan-morris]

Absolutely - happy to look further into it. Thanks for the possible versions!