Skip to content

Disable metaflow's logger #2429

@AntonisAgap

Description

@AntonisAgap

Hello metaflow team!
First of all I want to thank you for this awesome tool!

I'm posting this issue to ask if there is a way to disable metaflow's logging. I see that metaflow will catch all system's message and output them using click.secho. This causes the following issue on our part. We capture all the logs in Splunk which expects a json output format. However metaflow will append a string to the output and "pollutes" the message.

Here's an example piece of code:

from metaflow import FlowSpec, step
import structlog


def test_method(logger):
    logger.info("method called", step_method_name="test_method")


structlog.configure(processors=[structlog.processors.JSONRenderer()])
_logger = structlog.get_logger()

class MyFlow(FlowSpec):

    @step
    def start(self):
        _logger.info("flow_started", flow_step_name="start")
        test_method(_logger)
        self.next(self.end)

    @step
    def end(self):
        _logger.error("flow_ended", status="success")


def run() -> None:
    MyFlow()


if __name__ == "__main__":
    run()

which will output:

2025-05-26 18:51:15.530 Workflow starting (run-id 1748278275528814): 2025-05-26 18:51:15.545 [1748278275528814/start/1 (pid 91978)] Task is starting. 2025-05-26 18:51:15.799 [1748278275528814/start/1 (pid 91978)] {"flow_step_name": "start", "event": "flow_started"} 2025-05-26 18:51:15.832 [1748278275528814/start/1 (pid 91978)] {"step_method_name": "test_method", "event": "method called"} 2025-05-26 18:51:15.833 [1748278275528814/start/1 (pid 91978)] Task finished successfully. 2025-05-26 18:51:15.839 [1748278275528814/end/2 (pid 91981)] Task is starting. 2025-05-26 18:51:16.108 [1748278275528814/end/2 (pid 91981)] {"status": "success", "event": "flow_ended"} 2025-05-26 18:51:16.142 [1748278275528814/end/2 (pid 91981)] Task finished successfully. 2025-05-26 18:51:16.144 Done!

As you can see instead of having {"flow_step_name": "start", "event": "flow_started"}, the output is: 2025-05-26 18:51:15.799 [1748278275528814/start/1 (pid 91978)] {"flow_step_name": "start", "event": "flow_started"}.

Is there a way to output only info from my logger?

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions