Feature request: Late-bound context for buffered logs (apply current `append_keys` state at flush)

[CaptainDriftwood]

Use case

I’d like the Powertools Logger to support a mode where buffered log messages are enriched at flush-time using the current logger context (e.g., keys added later via append_keys / append_context_keys), rather than capturing context only at the moment each log call is made.

In other words: late-bound / deferred context enrichment for buffered logs.

---

Problem

In Lambda handlers, we often start logging immediately, but some high-value context only becomes available after initial parsing / I/O (e.g., dataset GUID, feed name, customer ID, document ID, correlation IDs derived from payload, etc.).

Today, if we log early and only later call append_keys(...), earlier log lines won’t include the new keys. The workaround is to either:

• delay logging (poor DX), or
• implement our own buffering wrapper and re-emit logs later (reinventing buffering + losing original timestamps unless we add custom fields).

We’d like to keep logging naturally throughout execution while still guaranteeing that all buffered lines include the final, complete context.

Solution/User Experience

Desired behavior

When log buffering is enabled, provide an option such that:

• logger.info/debug/... records are buffered
• at flush_buffer() / end-of-handler flush, each buffered record is emitted and merged with the current logger context state
  • i.e., keys appended after the log call are included in those earlier buffered records

Example:

from aws_lambda_powertools import Logger
from aws_lambda_powertools.logging import LoggerBufferConfig

logger = Logger(buffer_config=LoggerBufferConfig(max_bytes=1024 * 1024))
logger.info("start")  # buffered

# ... later we discover more context ...
logger.append_keys(dataset_guid="1234", feed="XYZ")

logger.info("parsed payload")  # buffered

logger.flush_buffer()

# EXPECTED: both "start" and "parsed payload" include dataset_guid/feed

Proposed API (one possible direction)

Add a buffer configuration option, for example:

• LoggerBufferConfig(..., enrich_on_flush=True)
  • When True, buffered entries do not snapshot context at write-time; instead they are enriched with current context at flush-time.

Or an explicit new buffering strategy:

• LoggerBufferConfig(..., mode="flush_enrich_current_context")

Alternative option:

• Allow a callable context provider:
  • Logger(buffer_config=..., buffer_context_provider=lambda: logger.get_current_keys())

---

Implementation notes

A couple approaches that could work:

1. Store log records + per-call extra only, and apply logger context in the formatter/handler at flush-time.
2. Store a reference to the context dict (or a resolver), rather than copying it, so that current state is used when formatting.
3. Provide a context manager/decorator for:
   • with logger.defer_logs(enrich_on_flush=True): ...
   • auto-flush at context exit (success or error)

---

Edge cases / expectations

• If a key is overwritten later via append_keys, flushed buffered records should reflect the latest value.
• Per-call extra/kwargs passed to logger.info(..., **kwargs) should override global context keys (or vice versa — but define it clearly).
• If the function errors or times out before flush, buffered logs may be lost (acceptable; same trade-off as buffering today).
• Thread-safety / async: behavior should be documented (Lambda is typically single-threaded per invocation, but users do run threads).

---

Why this belongs in Powertools

Powertools already supports both:

• structured context (append_keys, append_context_keys)
• log buffering + manual flush / flush-on-error patterns

This request bridges the two so users don’t have to roll a custom “deferred logger” wrapper just to ensure full context coverage.

Alternative solutions

Acknowledgment

• This feature request meets Powertools for AWS Lambda (Python) Tenets
• Should this be considered in other Powertools for AWS Lambda languages? i.e. Java, TypeScript, and .NET

[dreamorosi]

Hi @CaptainDriftwood, thank you for opening this feature request and including all the details.

I understand the use case however I am on the fence on whether we should add this or not. My main concern is around the positioning of the feature. Right now buffering is pretty straightforward: when enabled, logs below a certain level are buffered rather than emitted. That's it.

If we were to add this, even with a separate opt-in flag, there's a chance that the feature could become confusing for some - not to mention the internal implementation - as shown by the pretty significant edge cases / tradeoffs.

For example, the "If a key is overwritten later via append_keys, flushed buffered records should reflect the latest value." is quite significant and complicates the mental model quite a bit.

From an operations perspective we see a log as an entry with a discrete set of data emitted at a specific point in time. Manipulating the data after the fact breaks this contract and could complicate troubleshooting.

I don't have a good alternative at this time beyond logging the data you need later on in the function and relying on correlation IDs (i.e. request ID) to have an enriched view of the request despite the single entry not having the keys - but for now I'm unsure about this feature request.

Before making a decision I'd like to hear other opinions from other maintainers as well as get an idea of customer demand since the implementation is non-trivial.