s3 client certificate errors on windows

[aagmv]

Describe the bug

I'm getting the following errors from several concurrent s3 get_object requests on windows

DispatchFailure(DispatchFailure { source: ConnectorError { kind: Io, source: hyper::Error(Connect, Custom { kind: Other, error: Custom { kind: InvalidData, error: InvalidCertificate(UnknownIssue
r) } }), connection: Unknown } })

This is kind of surprising since dozens of previous requests worked fine just seconds earlier and I assume the Client (we're using the default configuration) uses connection pooling and does retries so at that point I don't expect it to make new TLS connections.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

The requests succeed

Current Behavior

Certificate errors prevent downloads to complete during system startup.

Reproduction Steps

I have not observed this issue when testing locally, so I don't have a simple reproducer.

Possible Solution

That would depend on the root-cause.

• if http connection pooling worked as expected then presumably the client wouldn't need to make new TLS handshakes that could fail
• if the retry default configuration isn't applied as expected maybe that needs fixing
• if there are issues with the cert loading then maybe we need to bake certs into our application build
• ....

Additional Information/Context

This happens on windows server 2025 EC2 instances in a rust program started early during instance initialization via userdata. The failures are intermittend, some instances launch fine, others see those errors and then cannot complete their lifecycle.
Additionally we're using the S3 gateway endpoint in the VPC.

We're building the SDK without default features so that ring gets used instead of aws-lc

Version

├── aws-config v1.8.3
│   ├── aws-credential-types v1.2.4
│   │   ├── aws-smithy-async v1.2.5
│   │   ├── aws-smithy-runtime-api v1.8.5
│   │   │   ├── aws-smithy-async v1.2.5 (*)
│   │   │   ├── aws-smithy-types v1.3.2
│   │   ├── aws-smithy-types v1.3.2 (*)
│   ├── aws-runtime v1.5.9
│   │   ├── aws-credential-types v1.2.4 (*)
│   │   ├── aws-sigv4 v1.3.3
│   │   │   ├── aws-credential-types v1.2.4 (*)
│   │   │   ├── aws-smithy-eventstream v0.60.10
│   │   │   │   ├── aws-smithy-types v1.3.2 (*)
│   │   │   ├── aws-smithy-http v0.62.2
│   │   │   │   ├── aws-smithy-eventstream v0.60.10 (*)
│   │   │   │   ├── aws-smithy-runtime-api v1.8.5 (*)
│   │   │   │   ├── aws-smithy-types v1.3.2 (*)
│   │   │   ├── aws-smithy-runtime-api v1.8.5 (*)
│   │   │   ├── aws-smithy-types v1.3.2 (*)
│   │   ├── aws-smithy-async v1.2.5 (*)
│   │   ├── aws-smithy-eventstream v0.60.10 (*)
│   │   ├── aws-smithy-http v0.62.2 (*)
│   │   ├── aws-smithy-runtime v1.8.5
│   │   │   ├── aws-smithy-async v1.2.5 (*)
│   │   │   ├── aws-smithy-http v0.62.2 (*)
│   │   │   ├── aws-smithy-http-client v1.0.6
│   │   │   │   ├── aws-smithy-async v1.2.5 (*)
│   │   │   │   ├── aws-smithy-runtime-api v1.8.5 (*)
│   │   │   │   ├── aws-smithy-types v1.3.2 (*)
│   │   │   ├── aws-smithy-observability v0.1.3
│   │   │   │   └── aws-smithy-runtime-api v1.8.5 (*)
│   │   │   ├── aws-smithy-runtime-api v1.8.5 (*)
│   │   │   ├── aws-smithy-types v1.3.2 (*)
│   │   ├── aws-smithy-runtime-api v1.8.5 (*)
│   │   ├── aws-smithy-types v1.3.2 (*)
│   │   ├── aws-types v1.3.8
│   │   │   ├── aws-credential-types v1.2.4 (*)
│   │   │   ├── aws-smithy-async v1.2.5 (*)
│   │   │   ├── aws-smithy-runtime-api v1.8.5 (*)
│   │   │   ├── aws-smithy-types v1.3.2 (*)
│   ├── aws-sdk-sts v1.80.0
│   │   ├── aws-credential-types v1.2.4 (*)
│   │   ├── aws-runtime v1.5.9 (*)
│   │   ├── aws-smithy-async v1.2.5 (*)
│   │   ├── aws-smithy-http v0.62.2 (*)
│   │   ├── aws-smithy-json v0.61.4
│   │   │   └── aws-smithy-types v1.3.2 (*)
│   │   ├── aws-smithy-query v0.60.7
│   │   │   ├── aws-smithy-types v1.3.2 (*)
│   │   ├── aws-smithy-runtime v1.8.5 (*)
│   │   ├── aws-smithy-runtime-api v1.8.5 (*)
│   │   ├── aws-smithy-types v1.3.2 (*)
│   │   ├── aws-smithy-xml v0.60.10
│   │   ├── aws-types v1.3.8 (*)
│   ├── aws-smithy-async v1.2.5 (*)
│   ├── aws-smithy-http v0.62.2 (*)
│   ├── aws-smithy-json v0.61.4 (*)
│   ├── aws-smithy-runtime v1.8.5 (*)
│   ├── aws-smithy-runtime-api v1.8.5 (*)
│   ├── aws-smithy-types v1.3.2 (*)
│   ├── aws-types v1.3.8 (*)
├── aws-sdk-autoscaling v1.88.0
│   ├── aws-credential-types v1.2.4 (*)
│   ├── aws-runtime v1.5.9 (*)
│   ├── aws-smithy-async v1.2.5 (*)
│   ├── aws-smithy-http v0.62.2 (*)
│   ├── aws-smithy-json v0.61.4 (*)
│   ├── aws-smithy-query v0.60.7 (*)
│   ├── aws-smithy-runtime v1.8.5 (*)
│   ├── aws-smithy-runtime-api v1.8.5 (*)
│   ├── aws-smithy-types v1.3.2 (*)
│   ├── aws-smithy-xml v0.60.10 (*)
│   ├── aws-types v1.3.8 (*)
├── aws-sdk-s3 v1.100.0
│   ├── aws-credential-types v1.2.4 (*)
│   ├── aws-runtime v1.5.9 (*)
│   ├── aws-sigv4 v1.3.3 (*)
│   ├── aws-smithy-async v1.2.5 (*)
│   ├── aws-smithy-checksums v0.63.5
│   │   ├── aws-smithy-http v0.62.2 (*)
│   │   ├── aws-smithy-types v1.3.2 (*)
│   ├── aws-smithy-eventstream v0.60.10 (*)
│   ├── aws-smithy-http v0.62.2 (*)
│   ├── aws-smithy-json v0.61.4 (*)
│   ├── aws-smithy-runtime v1.8.5 (*)
│   ├── aws-smithy-runtime-api v1.8.5 (*)
│   ├── aws-smithy-types v1.3.2 (*)
│   ├── aws-smithy-xml v0.60.10 (*)
│   ├── aws-types v1.3.8 (*)
│   ├── aws-config v1.8.3 (*)
│   ├── aws-sdk-s3 v1.100.0 (*)
├── aws-config v1.8.3 (*)
├── aws-sdk-s3 v1.100.0 (*)
├── aws-config v1.8.3 (*)
├── aws-sdk-autoscaling v1.88.0 (*)
├── aws-sdk-s3 v1.100.0 (*)
├── aws-smithy-runtime v1.8.5 (*)

Environment details (OS name and version, etc.)

Windows Server 2025

Logs

No response

[aagmv]

I've implemented manual retries around the get_object call and those do help. So either the built-in retries aren't working properly or I need to increase the retry count.

[2025-08-05T13:38:58.116049Z INFO  s3_downloader] done [redacted]
[2025-08-05T13:38:58.116118Z INFO  s3_downloader] Downloading [redacted]
[2025-08-05T13:38:58.537928Z INFO  s3_downloader] done [redacted]
[2025-08-05T13:38:58.537992Z INFO  s3_downloader] Downloading [redacted]
[2025-08-05T13:38:58.844330Z INFO  s3_downloader] done [redacted]
[2025-08-05T13:38:59.272269Z INFO  s3_downloader] done [redacted]
[2025-08-05T13:39:31.033100Z INFO  manager::machine::aws] Lifecycle heartbeat sent
[2025-08-05T13:39:36.992581Z WARN  s3_downloader] Failed to get object [redacted] at offset 23555211264; retry 1
[2025-08-05T13:39:37.143634Z WARN  s3_downloader] Failed to get object [redacted] at offset 23924310016; retry 1
[2025-08-05T13:39:41.090056Z WARN  s3_downloader] Failed to get object [redacted] at offset 25300041728; retry 1
[2025-08-05T13:39:42.190632Z WARN  s3_downloader] Failed to get object [redacted] at offset 25635586048; retry 1
[2025-08-05T13:39:44.267677Z WARN  s3_downloader] Failed to get object [redacted] at offset 25300041728; retry 2
[2025-08-05T13:39:45.995917Z WARN  s3_downloader] Failed to get object [redacted] at offset 25635586048; retry 2
[2025-08-05T13:39:55.474780Z WARN  s3_downloader] Failed to get object [redacted] at offset 32312918016; retry 1
[2025-08-05T13:40:11.210518Z INFO  manager::machine::aws] Lifecycle heartbeat sent
[2025-08-05T13:40:45.263263Z WARN  s3_downloader] Failed to get object [redacted] at offset 57243860992; retry 1
[2025-08-05T13:40:45.759468Z WARN  s3_downloader] Failed to get object [redacted] at offset 57847840768; retry 1
[2025-08-05T13:40:53.878851Z INFO  s3_downloader] Downloaded 64000775197 bytes in 123 seconds (495.27119538596156 MiB/sec)

[landonxjames]

Could you potentially enable tracing logs for the SDK so we can see what the logs look like when you are encountering the error?

[aagmv]

tracing produced gigabytes of logs that were a bit too unwieldy to analyze quickly. I've tried debug logging instead, which is manageable.

Do you want trace logging for a particular set of modules?

log excerpt

[2025-08-06T15:13:21.920493Z DEBUG aws_smithy_runtime::client::orchestrator] finally_attempt; attempt=2
[2025-08-06T15:13:21.920577Z DEBUG aws_smithy_runtime::client::retries::strategy::standard] attempt #2 failed with RetryIndicated(RetryableError { kind: TransientError, retry_after: None }); retrying after 72.562312ms
[2025-08-06T15:13:21.920606Z DEBUG aws_smithy_runtime::client::orchestrator] delaying for 72.562312ms
[2025-08-06T15:13:21.927064Z DEBUG aws_smithy_runtime::client::orchestrator] encountered orchestrator error; halting
[2025-08-06T15:13:21.927087Z DEBUG aws_smithy_runtime::client::orchestrator] finally_attempt; attempt=2
[2025-08-06T15:13:21.927116Z DEBUG aws_smithy_runtime::client::retries::strategy::standard] attempt #2 failed with RetryIndicated(RetryableError { kind: TransientError, retry_after: None }); retrying after 57.999537ms
[2025-08-06T15:13:21.927132Z DEBUG aws_smithy_runtime::client::orchestrator] delaying for 57.999537ms
[2025-08-06T15:13:21.945742Z DEBUG aws_sdk_s3::operation::get_object] S3.GetObject; rpc.service="S3" rpc.method="GetObject" sdk_invocation_id=5685679 rpc.system="aws-api"
[2025-08-06T15:13:21.945836Z DEBUG aws_smithy_runtime::client::orchestrator] timeout settings for this operation: TimeoutConfig { connect_timeout: Set(3.1s), read_timeout: Disabled, operation_timeout: Disabled, operation_attempt_timeout: Disabled }
[2025-08-06T15:13:21.945906Z DEBUG aws_smithy_runtime_api::client::interceptors::context] entering 'serialization' phase
[2025-08-06T15:13:21.945957Z DEBUG aws_smithy_runtime_api::client::interceptors::context] entering 'before transmit' phase
[2025-08-06T15:13:21.945998Z DEBUG aws_smithy_runtime::client::retries::strategy::standard] no client rate limiter configured, so no token is required for the initial request.
[2025-08-06T15:13:21.946026Z DEBUG aws_smithy_runtime::client::orchestrator] retry strategy has OKed initial request
[2025-08-06T15:13:21.946037Z DEBUG aws_smithy_runtime::client::orchestrator] beginning attempt #1
[2025-08-06T15:13:21.946045Z DEBUG aws_smithy_runtime::client::orchestrator] try_attempt; attempt=1
[2025-08-06T15:13:21.946060Z DEBUG aws_sdk_s3::endpoint_auth] resolving endpoint for auth scheme selection endpoint_params=EndpointResolverParams { inner: TypeErasedBox[!Clone]:Params { bucket: Some("[redacted]"), region: Some("eu-north-1"), use_fips: false, use_dual_stack: false, endpoint: None, force_path_style: false, accelerate: false, use_global_endpoint: false, use_object_lambda_endpoint: None, key: Some("[redacted]"), prefix: None, copy_source: None, disable_access_points: None, disable_multi_region_access_points: false, use_arn_region: None, use_s3_express_control_endpoint: None, disable_s3_express_session_auth: None }, property: {} }
[2025-08-06T15:13:21.946118Z DEBUG aws_smithy_runtime::client::identity::cache::lazy] loaded identity from cache buffer_time=10s cached_expiration=Some(SystemTime { intervals: 133989904030000000 }) now=SystemTime { intervals: 133989668019461149 }
[2025-08-06T15:13:21.946134Z DEBUG aws_smithy_runtime::client::orchestrator::endpoints] resolving endpoint endpoint_params=EndpointResolverParams { inner: TypeErasedBox[!Clone]:Params { bucket: Some("[redacted]"), region: Some("eu-north-1"), use_fips: false, use_dual_stack: false, endpoint: None, force_path_style: false, accelerate: false, use_global_endpoint: false, use_object_lambda_endpoint: None, key: Some("[redacted]"), prefix: None, copy_source: None, disable_access_points: None, disable_multi_region_access_points: false, use_arn_region: None, use_s3_express_control_endpoint: None, disable_s3_express_session_auth: None }, property: {TypeId(0x5b0d7eed5b2e3ca9c6d21b62d8bdc018): TypeErasedBox[!Clone]:Identity { data: Credentials { provider_name: "IMDSv2", access_key_id: "[redacted]", secret_access_key: "** redacted **", expires_after: "2025-08-06T21:46:43Z" }, expiration: Some(SystemTime { intervals: 133989904030000000 }) }} }
[2025-08-06T15:13:21.946171Z DEBUG aws_smithy_runtime::client::orchestrator::endpoints] will apply endpoint Endpoint { url: "https://[redacted].s3.eu-north-1.amazonaws.com", headers: {}, properties: {"authSchemes": Array([Object({"signingName": String("s3"), "name": String("sigv4"), "signingRegion": String("eu-north-1"), "disableDoubleEncoding": Bool(true)})])} } endpoint_prefix=None
[2025-08-06T15:13:21.946271Z DEBUG aws_smithy_runtime_api::client::interceptors::context] entering 'transmit' phase
[2025-08-06T15:13:21.966188Z DEBUG aws_smithy_runtime_api::client::interceptors::context] entering 'before deserialization' phase
[2025-08-06T15:13:21.966252Z DEBUG aws_smithy_runtime_api::client::interceptors::context] entering 'deserialization' phase
[2025-08-06T15:13:21.966262Z DEBUG aws_sdk_s3::operation::get_object] extended_request_id=Some("[redacted]")
[2025-08-06T15:13:21.966270Z DEBUG aws_sdk_s3::operation::get_object] request_id=Some("[redacted]")
[2025-08-06T15:13:21.966298Z DEBUG aws_smithy_runtime_api::client::interceptors::context] entering 'after deserialization' phase
[2025-08-06T15:13:21.966310Z DEBUG aws_smithy_runtime::client::orchestrator] finally_attempt; attempt=1
[2025-08-06T15:13:21.966327Z DEBUG aws_smithy_runtime::client::retries::strategy::standard] attempt #1 classified as NoActionIndicated, not retrying
[2025-08-06T15:13:21.966333Z DEBUG aws_smithy_runtime::client::orchestrator] a retry is either unnecessary or not possible, exiting attempt loop
[2025-08-06T15:13:21.991329Z DEBUG aws_smithy_runtime::client::orchestrator] beginning attempt #3
[2025-08-06T15:13:21.991351Z DEBUG aws_smithy_runtime::client::orchestrator] try_attempt; attempt=3
[2025-08-06T15:13:21.991367Z DEBUG aws_sdk_s3::endpoint_auth] resolving endpoint for auth scheme selection endpoint_params=EndpointResolverParams { inner: TypeErasedBox[!Clone]:Params { bucket: Some("[redacted]"), region: Some("eu-north-1"), use_fips: false, use_dual_stack: false, endpoint: None, force_path_style: false, accelerate: false, use_global_endpoint: false, use_object_lambda_endpoint: None, key: Some("[redacted]"), prefix: None, copy_source: None, disable_access_points: None, disable_multi_region_access_points: false, use_arn_region: None, use_s3_express_control_endpoint: None, disable_s3_express_session_auth: None }, property: {TypeId(0x5b0d7eed5b2e3ca9c6d21b62d8bdc018): TypeErasedBox[!Clone]:Identity { data: Credentials { provider_name: "IMDSv2", access_key_id: "[redacted]", secret_access_key: "** redacted **", expires_after: "2025-08-06T21:46:43Z" }, expiration: Some(SystemTime { intervals: 133989904030000000 }) }} }
[2025-08-06T15:13:21.991424Z DEBUG aws_smithy_runtime::client::identity::cache::lazy] loaded identity from cache buffer_time=10s cached_expiration=Some(SystemTime { intervals: 133989904030000000 }) now=SystemTime { intervals: 133989668019914210 }
[2025-08-06T15:13:21.991436Z DEBUG aws_smithy_runtime::client::orchestrator::endpoints] resolving endpoint endpoint_params=EndpointResolverParams { inner: TypeErasedBox[!Clone]:Params { bucket: Some("[redacted]"), region: Some("eu-north-1"), use_fips: false, use_dual_stack: false, endpoint: None, force_path_style: false, accelerate: false, use_global_endpoint: false, use_object_lambda_endpoint: None, key: Some("[redacted]"), prefix: None, copy_source: None, disable_access_points: None, disable_multi_region_access_points: false, use_arn_region: None, use_s3_express_control_endpoint: None, disable_s3_express_session_auth: None }, property: {TypeId(0x5b0d7eed5b2e3ca9c6d21b62d8bdc018): TypeErasedBox[!Clone]:Identity { data: Credentials { provider_name: "IMDSv2", access_key_id: "[redacted]", secret_access_key: "** redacted **", expires_after: "2025-08-06T21:46:43Z" }, expiration: Some(SystemTime { intervals: 133989904030000000 }) }} }
[2025-08-06T15:13:21.991464Z DEBUG aws_smithy_runtime::client::orchestrator::endpoints] will apply endpoint Endpoint { url: "https://[redacted].s3.eu-north-1.amazonaws.com", headers: {}, properties: {"authSchemes": Array([Object({"signingName": String("s3"), "name": String("sigv4"), "disableDoubleEncoding": Bool(true), "signingRegion": String("eu-north-1")})])} } endpoint_prefix=None
[2025-08-06T15:13:21.991576Z DEBUG aws_smithy_runtime_api::client::interceptors::context] entering 'transmit' phase
[2025-08-06T15:13:21.995124Z DEBUG aws_smithy_runtime::client::orchestrator] beginning attempt #3
[2025-08-06T15:13:21.995191Z DEBUG aws_smithy_runtime::client::orchestrator] try_attempt; attempt=3
[2025-08-06T15:13:21.995218Z DEBUG aws_sdk_s3::endpoint_auth] resolving endpoint for auth scheme selection endpoint_params=EndpointResolverParams { inner: TypeErasedBox[!Clone]:Params { bucket: Some("[redacted]"), region: Some("eu-north-1"), use_fips: false, use_dual_stack: false, endpoint: None, force_path_style: false, accelerate: false, use_global_endpoint: false, use_object_lambda_endpoint: None, key: Some("[redacted]"), prefix: None, copy_source: None, disable_access_points: None, disable_multi_region_access_points: false, use_arn_region: None, use_s3_express_control_endpoint: None, disable_s3_express_session_auth: None }, property: {TypeId(0x5b0d7eed5b2e3ca9c6d21b62d8bdc018): TypeErasedBox[!Clone]:Identity { data: Credentials { provider_name: "IMDSv2", access_key_id: "[redacted]", secret_access_key: "** redacted **", expires_after: "2025-08-06T21:46:43Z" }, expiration: Some(SystemTime { intervals: 133989904030000000 }) }} }
[2025-08-06T15:13:21.995303Z DEBUG aws_smithy_runtime::client::identity::cache::lazy] loaded identity from cache buffer_time=10s cached_expiration=Some(SystemTime { intervals: 133989904030000000 }) now=SystemTime { intervals: 133989668019952998 }
[2025-08-06T15:13:21.995318Z DEBUG aws_smithy_runtime::client::orchestrator::endpoints] resolving endpoint endpoint_params=EndpointResolverParams { inner: TypeErasedBox[!Clone]:Params { bucket: Some("[redacted]"), region: Some("eu-north-1"), use_fips: false, use_dual_stack: false, endpoint: None, force_path_style: false, accelerate: false, use_global_endpoint: false, use_object_lambda_endpoint: None, key: Some("[redacted]"), prefix: None, copy_source: None, disable_access_points: None, disable_multi_region_access_points: false, use_arn_region: None, use_s3_express_control_endpoint: None, disable_s3_express_session_auth: None }, property: {TypeId(0x5b0d7eed5b2e3ca9c6d21b62d8bdc018): TypeErasedBox[!Clone]:Identity { data: Credentials { provider_name: "IMDSv2", access_key_id: "[redacted]", secret_access_key: "** redacted **", expires_after: "2025-08-06T21:46:43Z" }, expiration: Some(SystemTime { intervals: 133989904030000000 }) }} }
[2025-08-06T15:13:21.995342Z DEBUG aws_smithy_runtime::client::orchestrator::endpoints] will apply endpoint Endpoint { url: "https://[redacted].s3.eu-north-1.amazonaws.com", headers: {}, properties: {"authSchemes": Array([Object({"signingName": String("s3"), "name": String("sigv4"), "disableDoubleEncoding": Bool(true), "signingRegion": String("eu-north-1")})])} } endpoint_prefix=None
[2025-08-06T15:13:21.995458Z DEBUG aws_smithy_runtime_api::client::interceptors::context] entering 'transmit' phase
[2025-08-06T15:13:21.997223Z DEBUG aws_smithy_runtime::client::orchestrator] encountered orchestrator error; halting
[2025-08-06T15:13:21.997243Z DEBUG aws_smithy_runtime::client::orchestrator] finally_attempt; attempt=3
[2025-08-06T15:13:21.997266Z DEBUG aws_smithy_runtime::client::retries::strategy::standard] not retrying because we are out of attempts attempts=3 max_attempts=3
[2025-08-06T15:13:21.997272Z DEBUG aws_smithy_runtime::client::orchestrator] a retry is either unnecessary or not possible, exiting attempt loop
[2025-08-06T15:13:21.997348Z WARN  s3_downloader] Failed to get object [redacted] at offset 20266876928; retry 1

[2025-08-06T15:13:21.927116Z DEBUG aws_smithy_runtime::client::retries::strategy::standard] attempt #2 failed with RetryIndicated(RetryableError { kind: TransientError, retry_after: None }); retrying after 57.999537ms
[2025-08-06T15:13:21.997266Z DEBUG aws_smithy_runtime::client::retries::strategy::standard] not retrying because we are out of attempts attempts=3 max_attempts=3

So if I read this correctly the built-in retry defaults do something but they're too aggressive - it only waited a few milliseconds - to paper over the UnknownIssuer error.

With a handrolled retry of 2 seconds things have worked so far. I'll try adjusting the retrystrategy settings then.

[aagmv]

I have the suspicion that a VPC S3 gateway endpoint resolves to multiple IP addresses and the server listening on one of the IP addresses doesn't work or at least triggers TLS handshake failures while the others do. And the retries paper over that as long as they they end up with a different IP address.

[2025-08-06T17:22:53.093592Z DEBUG hyper::client::connect::dns] resolving host="[bucket].s3.eu-north-1.amazonaws.com"
[2025-08-06T17:22:53.094056Z DEBUG hyper::client::connect::http] connecting to 3.5.216.143:443
[2025-08-06T17:22:53.095583Z DEBUG hyper::client::connect::http] connected to 3.5.216.143:443
[2025-08-06T17:22:53.095611Z DEBUG rustls::client::hs] No cached session for DnsName("[bucket].s3.eu-north-1.amazonaws.com")
[2025-08-06T17:22:53.095676Z DEBUG rustls::client::hs] Not resuming any session
[2025-08-06T17:22:53.096988Z DEBUG rustls::client::hs] Using ciphersuite TLS13_AES_128_GCM_SHA256
[2025-08-06T17:22:53.097007Z DEBUG rustls::client::tls13] Not resuming
[2025-08-06T17:22:53.097110Z DEBUG rustls::client::tls13] TLS1.3 encrypted extensions: [ServerNameAck, Protocols([ProtocolName(687474702f312e31)])]
[2025-08-06T17:22:53.097123Z DEBUG rustls::client::hs] ALPN protocol is Some(b"http/1.1")
[2025-08-06T17:22:53.097820Z DEBUG aws_smithy_runtime::client::orchestrator] encountered orchestrator error; halting

[2025-08-06T17:22:55.444766Z DEBUG hyper::client::connect::dns] resolving host="[bucket].s3.eu-north-1.amazonaws.com"
[2025-08-06T17:22:55.445158Z DEBUG hyper::client::connect::http] connecting to 3.5.216.143:443
[2025-08-06T17:22:55.462657Z DEBUG hyper::client::connect::http] connected to 3.5.216.143:443
[2025-08-06T17:22:55.462708Z DEBUG rustls::client::hs] No cached session for DnsName("[bucket].s3.eu-north-1.amazonaws.com")
[2025-08-06T17:22:55.462788Z DEBUG rustls::client::hs] Not resuming any session
[2025-08-06T17:22:55.463956Z DEBUG rustls::client::hs] Using ciphersuite TLS13_AES_128_GCM_SHA256
[2025-08-06T17:22:55.463975Z DEBUG rustls::client::tls13] Not resuming
[2025-08-06T17:22:55.464076Z DEBUG rustls::client::tls13] TLS1.3 encrypted extensions: [ServerNameAck, Protocols([ProtocolName(687474702f312e31)])]
[2025-08-06T17:22:55.464086Z DEBUG rustls::client::hs] ALPN protocol is Some(b"http/1.1")
[2025-08-06T17:22:55.465084Z DEBUG aws_smithy_runtime::client::orchestrator] encountered orchestrator error; halting


[2025-08-06T17:23:10.114697Z DEBUG hyper::client::connect::dns] resolving host="[bucket].s3.eu-north-1.amazonaws.com"
[2025-08-06T17:23:10.115201Z DEBUG hyper::client::connect::http] connecting to 16.12.9.14:443
[2025-08-06T17:23:10.115597Z DEBUG hyper::client::connect::http] connected to 16.12.9.14:443
[2025-08-06T17:23:10.115626Z DEBUG rustls::client::hs] No cached session for DnsName("[bucket].s3.eu-north-1.amazonaws.com")
[2025-08-06T17:23:10.115713Z DEBUG rustls::client::hs] Not resuming any session
[2025-08-06T17:23:10.116190Z DEBUG rustls::client::hs] Using ciphersuite TLS13_AES_128_GCM_SHA256
[2025-08-06T17:23:10.116230Z DEBUG rustls::client::tls13] Not resuming
[2025-08-06T17:23:10.116439Z DEBUG rustls::client::tls13] TLS1.3 encrypted extensions: [ServerNameAck, Protocols([ProtocolName(687474702f312e31)])]
[2025-08-06T17:23:10.116466Z DEBUG rustls::client::hs] ALPN protocol is Some(b"http/1.1")
[2025-08-06T17:23:10.117250Z DEBUG hyper::proto::h1::io] flushed 2306 bytes
[2025-08-06T17:23:10.131423Z DEBUG hyper::proto::h1::io] parsed 11 headers
[2025-08-06T17:23:10.131451Z DEBUG hyper::proto::h1::conn] incoming body is content-length (33554432 bytes)
[2025-08-06T17:23:10.131555Z TRACE aws_smithy_runtime::client::orchestrator] received response from service response=Response { status: StatusCode(206)

[aagmv]

After some debugging and help from AWS Support it looks like the Windows Server AMI are using doesn't have the AWS Root CAs installed in its trust store.
Adding them to TlsContext of the S3 client solved the problem.

[github-actions]

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.