python_template_launcher.go:37: Failed to initialize fluentd logger

[pbonito]

In which file did you encounter the issue?

python-docs-samples/dataflow/flex-templates/pipeline_with_dependencies

Did you change the file? If so, how?

Yes, used a different base image in docker file

Describe the issue

Jon start but I got following error:
2025/03/06 17:38:45.885541 python_template_launcher.go:37: Failed to initialize fluentd logger: failed to create log file /var/log/dataflow/template_launcher/runner-json.log: open /var/log/dataflow/template_launcher/runner-json.log: permission denied

python_template_launcher fails if running with non root user or an user that have no write access to /var/log

[iennae]

Thanks @pbonito. Can you share more details about your Dockerfile configuration? which base image are you using?

[pbonito]

This is an example, my real image is hardened and running with nonroot user and no access to /var/log.

FROM nvcr.io/nvidia/cuda:12.6.1-cudnn-devel-ubuntu24.04

ARG WORKDIR=/template
WORKDIR ${WORKDIR}

COPY requirements.txt .
COPY .py ./
RUN apt-get update
# Install Python and other system dependencies.
&& apt-get install -y --no-install-recommends software-properties-common
&& add-apt-repository ppa:deadsnakes/ppa -y
&& apt-get install -y --no-install-recommends
curl g++ python3.11-dev python3.11-venv apt-transport-https ca-certificates gnupg
&& rm -rf /var/lib/apt/lists/
&& update-alternatives --install /usr/bin/python python /usr/bin/python3.11 11
&& curl https://bootstrap.pypa.io/get-pip.py | python
# Install the pipeline requirements.
&& pip install --no-cache-dir -r requirements.txt
&& pip check

COPY --from=gcr.io/cloud-dataflow/v1beta3/beam_python3.11_sdk:2.61.0 /opt/apache/beam /opt/apache/beam
COPY --from=gcr.io/dataflow-templates-base/python311-template-launcher-base:latest /opt/google/dataflow/python_template_launcher /opt/google/dataflow/python_template_launcher

COPY main.py .
COPY pyproject.toml .
COPY setup.py .
COPY src src

RUN pip install .
ENV FLEX_TEMPLATE_PYTHON_PY_FILE="${WORKDIR}/main.py"
RUN pip freeze
RUN pip list

ENTRYPOINT [ "/opt/apache/beam/boot" ]

[iennae]

@pbonito thanks so much for the information. I can see the challenge with the current sample for custom Docker images. We're investigating what the right path is here. If you have a support contract with Google, please create an issue in the support console as well. Thank you!