Add environment variable to support backend storage tag enrichment #1006
Conversation
ViBiOh
changed the title
ViBiOh
force-pushed
the
aws-forwarder-v5-preparation
branch
from
aa79f3a to
f7fb1e1
Compare
ViBiOh
force-pushed
the
vibioh/AWSX-1566_storage_tag
branch
from
3994c02 to
e7004da
Compare
Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com>
Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com>
ViBiOh
force-pushed
the
vibioh/AWSX-1566_storage_tag
branch
from
e7004da to
c9ec9ae
Compare
Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com>
aws/logs_monitoring/README.md
Outdated
| ### Advanced (optional) | ||
| `DD_ENRICH_S3_TAGS` | ||
| : Instruct Datadog backend to enrich a log coming from a S3 bucket with the tag attached to this bucket. It's the equivalent behavior of `DD_FETCH_S3_TAG` but done after ingestion. This require Resource Collection to be enabled. Flag is enabled by default. |
There was a problem hiding this comment.
I'd rephrase with something like
True by default. Enables S3 tag enrichment at intake time for logs coming from S3 buckets. Equivalent to DD_FETCH_S3_TAG` once the logs are ingested in Datadog. This parameter requires Resource Collection to be enabled.
There was a problem hiding this comment.
"at intake time" is a bit ambiguous, before or after intake? because it impacts billing.
"S3 tag" is also ambiguous, "S3 Bucket tag" is precise.
There was a problem hiding this comment.
I think I feel strongly about having a link to our documentation for resource collection.
For the other parts I don't mind keeping them as is, the key part that we want the reader to understand is that the tags will appear in the logs after they're ingested by Datadog, meaning they don't pay for the volume of the log the tag contribute to
There was a problem hiding this comment.
Revamped it with Claude
|
|
||
| storage_tag = get_dd_storage_tag_header() | ||
| if storage_tag != "": | ||
| _HEADERS["DD-STORAGE-TAG"] = storage_tag |
There was a problem hiding this comment.
maybe set the header key as a constant (now that I comment that I'd argue the other should be too)
There was a problem hiding this comment.
It's only used in one place. I don't think it worth it. We'll do some back and forth in code for something that is used at only one place.
… metrics Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com>
Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com>
3 participants
What does this PR do?
This pull-request switch the tag enrichment done by the forwarder to logs-backend by default for both s3 and cloudwatch.
Motivation
Reduce ingestion payload size, s3 reduce cache burden and cost.
Testing Guidelines
Additional Notes
Types of changes
Check all that apply