-
Notifications
You must be signed in to change notification settings - Fork 1.8k
Description
Violation of Postel's Law and RFC9100 when handling the ETag HTTP header
Describe the bug
The AWS S3 Output fails to carry out AWS S3 MultipartUploads for S3 compatible services (for instance EMC ViPR/1.0) because it does not support the header name ETag to be specified in small letters against what is written in RFC 9100 which states that "HTTP header field names are case-insensitive".
To Reproduce
You need an S3 compatible service that uses small letters with header names. I'm dealing with EMC ViPR (behind my cloud provider of choice which is UpCloud and I will understand you have not heard of it). On top of that I'm running FluentBit manged by the FluentBit Kubernetes operator. Just define the S3 output with UploadChucksize:15728640 in the FluentBit YAML configuration file (or the ClusterOutput Kubernetes resource).
When uploading a Multipart you'll get this:
UploadPart http status=200
[2025/11/25 07:35:23] [error] [output:s3:s3.2] Could not find ETag in UploadPart response
[2025/11/25 07:35:23] [debug] [output:s3:s3.2] Raw UploadPart response: (null)
[2025/11/25 07:35:23] [debug] [output:s3:s3.2] Failed to upload file in upload_queue. Will not retry for 2 seconds
[2025/11/25 07:35:23] [engine] caught signal (SIGSEGV)
#0 0x7fa03dd55257 in __vfprintf_internal() at fprintf-process-arg.c:397
#1 0x7fa03dd75757 in __vsnprintf_internal() at f.c:114
#2 0x55730205d4a3 in vsnprintf() at _64-linux-gnu/bits/stdio2.h:68
#3 0x55730205d4a3 in flb_log_construct() at src/flb_log.c:747
#4 0x55730205d739 in flb_log_print() at src/flb_log.c:810
#5 0x5573021d5eaf in get_upload() at plugins/out_s3/s3.c:1501
#6 0x5573021da367 in cb_s3_flush() at plugins/out_s3/s3.c:2187
#7 0x5573027036a6 in co_init() at lib/monkey/deps/flb_libco/amd64.c:117
level=error time=2025-11-25T07:35:23Z msg="Failure during the run time of fluent-bit" error="failed to run fluent-bit: signal: segmentation fault (core dumped)"
Expected behavior
I expect the MultipartUpload process to continue.
Screenshots
Rather than a screenshot I'll show the offending candidate in plugins/out_s3/s3_multipart.c:
/* gets the ETag value from response headers */
flb_sds_t get_etag(char *response, size_t size)
{
char *tmp;
int start;
int end;
int len;
int i = 0;
flb_sds_t etag;
if (response == NULL) {
return NULL;
}
tmp = strstr(response, "ETag:");
Notice that strstr(3) performs a case sensitive comparison (against what RFC 9110 states).
Your Environment
master branch at commit ID f4108db65e876..
Additional context
An easy fix would be to use strcasestr(3) but that may require (?) some harness in CMake so that it does not break the build on systems without GNU glibc.
In any case proprietary storage management systems often support S3 compatible HTTP interfaces and I presume some of them (not only EMC) use header names will small letters.
Thanks for your time reading this.