Skip to content

[core] Rename LogRecord/log_record to LogData/log_data #53

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: nhairs-str-to-object
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 16 additions & 1 deletion docs/changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,27 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [UNRELEASED]
## [4.0.0](https://github.com/nhairs/python-json-logger/compare/v3.3.3...v4.0.0) - UNRELEASED

### Added
- Support `DictConfigurator` prefixes for `rename_fields` and `static_fields`. [#45](https://github.com/nhairs/python-json-logger/pull/45)
- Allows using values like `ext://sys.stderr` in `fileConfig`/`dictConfig` value fields.

### Changed
- Rename `pythonjsonlogger.core.LogRecord` and `log_record` arguments to avoid confusion / overlapping with `logging.LogRecord`. [#38](https://github.com/nhairs/python-json-logger/issues/38)
- Affects arguments to `pythonjsonlogger.core.BaseJsonFormatter` (and any child classes).
- `serialize_log_record`
- `add_fields`
- `jsonify_log_record`
- `process_log_record`
- Note: functions referring to `log_record` have **not** had their function name changed.

### Removed
- Remove support for providing strings instead of objects when instantiating formatters. Instead use the `DictConfigurator` `ext://` prefix format when using `fileConfig`/`dictConfig`. [#47](https://github.com/nhairs/python-json-logger/issues/47)
- Affects `pythonjsonlogger.json.JsonFormatter`: `json_default`, `json_encoder`, `json_serializer`.
- Affects `pythonjsonlogger.orjson.OrjsonFormatter`: `json_default`.
- Affects `pythonjsonlogger.msgspec.MsgspecFormatter`: `json_default`.

Thanks @rubensa

## [3.3.0](https://github.com/nhairs/python-json-logger/compare/v3.2.1...v3.3.0) - 2025-03-06
Expand Down
23 changes: 17 additions & 6 deletions docs/cookbook.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,8 +32,8 @@ You can modify the `dict` of data that will be logged by overriding the `process

```python
class SillyFormatter(JsonFormatter):
def process_log_record(log_record):
new_record = {k[::-1]: v for k, v in log_record.items()}
def process_log_record(log_data):
new_record = {k[::-1]: v for k, v in log_data.items()}
return new_record
```

Expand Down Expand Up @@ -119,9 +119,9 @@ Another method would be to create a custom formatter class and override the `pro
## -----------------------------------------------------------------------------
# Reuse REQUEST_ID stuff from solution 2
class MyFormatter(JsonFormatter):
def process_log_record(self, log_record):
log_record["request_id"] = get_request_id()
return log_record
def process_log_record(self, log_data):
log_data["request_id"] = get_request_id()
return log_data

handler.setFormatter(MyFormatter())

Expand All @@ -148,6 +148,7 @@ formatters:
default:
"()": pythonjsonlogger.json.JsonFormatter
format: "%(asctime)s %(levelname)s %(name)s %(module)s %(funcName)s %(lineno)s %(message)s"
json_default: ext://logging_config.my_json_default
rename_fields:
"asctime": "timestamp"
"levelname": "status"
Expand Down Expand Up @@ -178,13 +179,23 @@ loggers:
propagate: no
```

You'll notice that we are using `ext://...` for the `static_fields`. This will load data from other modules such as the one below.
You'll notice that we are using `ext://...` for `json_default` and`static_fields`. This will load data from other modules such as the one below.

```python title="logging_config.py"
import importlib.metadata
import os


class Dummy:
pass


def my_json_default(obj: Any) -> Any:
if isinstance(obj, Dummy):
return "DUMMY"
return obj


def get_version_metadata():
# https://stackoverflow.com/a/78082532
version = importlib.metadata.version(PROJECT_NAME)
Expand Down
85 changes: 38 additions & 47 deletions src/pythonjsonlogger/core.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,10 @@

## Standard Library
from datetime import datetime, timezone
import importlib
import logging
import re
import sys
from typing import Optional, Union, Callable, List, Dict, Container, Any, Sequence
from typing import Optional, Union, List, Dict, Container, Any, Sequence

if sys.version_info >= (3, 10):
from typing import TypeAlias
Expand Down Expand Up @@ -72,31 +71,15 @@

## Type Aliases
## -----------------------------------------------------------------------------
OptionalCallableOrStr: TypeAlias = Optional[Union[Callable, str]]
"""Type alias"""
LogData: TypeAlias = Dict[str, Any]
"""Type alias

LogRecord: TypeAlias = Dict[str, Any]
"""Type alias"""
*Changed in 4.0*: renamed from `LogRecord` to `LogData`
"""


### FUNCTIONS
### ============================================================================
def str_to_object(obj: Any) -> Any:
"""Import strings to an object, leaving non-strings as-is.

Args:
obj: the object or string to process

*New in 3.1*
"""

if not isinstance(obj, str):
return obj

module_name, attribute_name = obj.rsplit(".", 1)
return getattr(importlib.import_module(module_name), attribute_name)


def merge_record_extra(
record: logging.LogRecord,
target: Dict,
Expand Down Expand Up @@ -135,7 +118,7 @@ class BaseJsonFormatter(logging.Formatter):

*Changed in 3.2*: `defaults` argument is no longer ignored.

*Added in UNRELEASED*: `exc_info_as_array` and `stack_info_as_array` options are added.
*Added in 4.0*: `exc_info_as_array` and `stack_info_as_array` options are added.
"""

_style: Union[logging.PercentStyle, str] # type: ignore[assignment]
Expand Down Expand Up @@ -269,11 +252,11 @@ def format(self, record: logging.LogRecord) -> str:
if record.stack_info and not message_dict.get("stack_info"):
message_dict["stack_info"] = self.formatStack(record.stack_info)

log_record: LogRecord = {}
self.add_fields(log_record, record, message_dict)
log_record = self.process_log_record(log_record)
log_data: LogData = {}
self.add_fields(log_data, record, message_dict)
log_data = self.process_log_record(log_data)

return self.serialize_log_record(log_record)
return self.serialize_log_record(log_data)

## JSON Formatter Specific Methods
## -------------------------------------------------------------------------
Expand Down Expand Up @@ -307,17 +290,19 @@ def parse(self) -> List[str]:

return []

def serialize_log_record(self, log_record: LogRecord) -> str:
"""Returns the final representation of the log record.
def serialize_log_record(self, log_data: LogData) -> str:
"""Returns the final representation of the data to be logged

Args:
log_record: the log record
log_data: the data

*Changed in 4.0*: `log_record` renamed to `log_data`
"""
return self.prefix + self.jsonify_log_record(log_record)
return self.prefix + self.jsonify_log_record(log_data)

def add_fields(
self,
log_record: Dict[str, Any],
log_data: Dict[str, Any],
record: logging.LogRecord,
message_dict: Dict[str, Any],
) -> None:
Expand All @@ -326,65 +311,71 @@ def add_fields(
This method can be overridden to implement custom logic for adding fields.

Args:
log_record: data that will be logged
log_data: data that will be logged
record: the record to extract data from
message_dict: dictionary that was logged instead of a message. e.g
`logger.info({"is_this_message_dict": True})`

*Changed in 4.0*: `log_record` renamed to `log_data`
"""
for field in self.defaults:
log_record[self._get_rename(field)] = self.defaults[field]
log_data[self._get_rename(field)] = self.defaults[field]

for field in self._required_fields:
log_record[self._get_rename(field)] = record.__dict__.get(field)
log_data[self._get_rename(field)] = record.__dict__.get(field)

for data_dict in [self.static_fields, message_dict]:
for key, value in data_dict.items():
log_record[self._get_rename(key)] = value
log_data[self._get_rename(key)] = value

merge_record_extra(
record,
log_record,
log_data,
reserved=self._skip_fields,
rename_fields=self.rename_fields,
)

if self.timestamp:
key = self.timestamp if isinstance(self.timestamp, str) else "timestamp"
log_record[self._get_rename(key)] = datetime.fromtimestamp(
log_data[self._get_rename(key)] = datetime.fromtimestamp(
record.created, tz=timezone.utc
)

if self.rename_fields_keep_missing:
for field in self.rename_fields.values():
if field not in log_record:
log_record[field] = None
if field not in log_data:
log_data[field] = None
return

def _get_rename(self, key: str) -> str:
return self.rename_fields.get(key, key)

# Child Methods
# ..........................................................................
def jsonify_log_record(self, log_record: LogRecord) -> str:
"""Convert this log record into a JSON string.
def jsonify_log_record(self, log_data: LogData) -> str:
"""Convert the log data into a JSON string.

Child classes MUST override this method.

Args:
log_record: the data to serialize
log_data: the data to serialize

*Changed in 4.0*: `log_record` renamed to `log_data`
"""
raise NotImplementedError()

def process_log_record(self, log_record: LogRecord) -> LogRecord:
"""Custom processing of the log record.
def process_log_record(self, log_data: LogData) -> LogData:
"""Custom processing of the data to be logged.

Child classes can override this method to alter the log record before it
is serialized.

Args:
log_record: incoming data
log_data: incoming data

*Changed in 4.0*: `log_record` renamed to `log_data`
"""
return log_record
return log_data

def formatException(self, ei) -> Union[str, list[str]]: # type: ignore
"""Format and return the specified exception information.
Expand Down
18 changes: 9 additions & 9 deletions src/pythonjsonlogger/json.py
Original file line number Diff line number Diff line change
Expand Up @@ -67,9 +67,9 @@ class JsonFormatter(core.BaseJsonFormatter):
def __init__(
self,
*args,
json_default: core.OptionalCallableOrStr = None,
json_encoder: core.OptionalCallableOrStr = None,
json_serializer: Union[Callable, str] = json.dumps,
json_default: Optional[Callable] = None,
json_encoder: Optional[Callable] = None,
json_serializer: Callable = json.dumps,
json_indent: Optional[Union[int, str]] = None,
json_ensure_ascii: bool = True,
**kwargs,
Expand All @@ -87,19 +87,19 @@ def __init__(
"""
super().__init__(*args, **kwargs)

self.json_default = core.str_to_object(json_default)
self.json_encoder = core.str_to_object(json_encoder)
self.json_serializer = core.str_to_object(json_serializer)
self.json_default = json_default
self.json_encoder = json_encoder
self.json_serializer = json_serializer
self.json_indent = json_indent
self.json_ensure_ascii = json_ensure_ascii
if not self.json_encoder and not self.json_default:
self.json_encoder = JsonEncoder
return

def jsonify_log_record(self, log_record: core.LogRecord) -> str:
"""Returns a json string of the log record."""
def jsonify_log_record(self, log_data: core.LogData) -> str:
"""Returns a json string of the log data."""
return self.json_serializer(
log_record,
log_data,
default=self.json_default,
cls=self.json_encoder,
indent=self.json_indent,
Expand Down
12 changes: 6 additions & 6 deletions src/pythonjsonlogger/msgspec.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
from __future__ import annotations

## Standard Library
from typing import Any
from typing import Any, Optional, Callable

## Installed

Expand Down Expand Up @@ -43,7 +43,7 @@ class MsgspecFormatter(core.BaseJsonFormatter):
def __init__(
self,
*args,
json_default: core.OptionalCallableOrStr = msgspec_default,
json_default: Optional[Callable] = msgspec_default,
**kwargs,
) -> None:
"""
Expand All @@ -54,10 +54,10 @@ def __init__(
"""
super().__init__(*args, **kwargs)

self.json_default = core.str_to_object(json_default)
self.json_default = json_default
self._encoder = msgspec.json.Encoder(enc_hook=self.json_default)
return

def jsonify_log_record(self, log_record: core.LogRecord) -> str:
"""Returns a json string of the log record."""
return self._encoder.encode(log_record).decode("utf8")
def jsonify_log_record(self, log_data: core.LogData) -> str:
"""Returns a json string of the log data."""
return self._encoder.encode(log_data).decode("utf8")
12 changes: 6 additions & 6 deletions src/pythonjsonlogger/orjson.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
from __future__ import annotations

## Standard Library
from typing import Any
from typing import Any, Optional, Callable

## Installed

Expand Down Expand Up @@ -45,7 +45,7 @@ class OrjsonFormatter(core.BaseJsonFormatter):
def __init__(
self,
*args,
json_default: core.OptionalCallableOrStr = orjson_default,
json_default: Optional[Callable] = orjson_default,
json_indent: bool = False,
**kwargs,
) -> None:
Expand All @@ -58,14 +58,14 @@ def __init__(
"""
super().__init__(*args, **kwargs)

self.json_default = core.str_to_object(json_default)
self.json_default = json_default
self.json_indent = json_indent
return

def jsonify_log_record(self, log_record: core.LogRecord) -> str:
"""Returns a json string of the log record."""
def jsonify_log_record(self, log_data: core.LogData) -> str:
"""Returns a json string of the log data."""
opt = orjson.OPT_NON_STR_KEYS
if self.json_indent:
opt |= orjson.OPT_INDENT_2

return orjson.dumps(log_record, default=self.json_default, option=opt).decode("utf8")
return orjson.dumps(log_data, default=self.json_default, option=opt).decode("utf8")
Loading