Change-over to go

Author: hhftechnology

Dockerfile

@@ -1,4 +1,43 @@
-FROM python:3.9-slim
+FROM golang:1.21-alpine AS builder
+
+WORKDIR /build
+
+# Copy go mod and sum files
+COPY go.mod go.sum ./
+
+# Download dependencies
+RUN go mod download
+
+# Copy source code
+COPY cmd/ ./cmd/
+
+# Build the application
+RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o traefik-log-processor ./cmd/main.go
+
+# Use a minimal alpine image for the final container
+FROM alpine:3.17
+
+# Add ca-certificates for any HTTPS connections
+RUN apk --no-cache add ca-certificates tzdata
+
 WORKDIR /app
-COPY process_logs.py .
-CMD ["python", "process_logs.py"]
+
+# Copy the binary from the builder stage
+COPY --from=builder /build/traefik-log-processor .
+
+# Copy default config
+COPY config.yaml .
+
+# Create directories for logs
+RUN mkdir -p /logs /output
+
+# Set user to non-root
+RUN addgroup -g 1000 appuser && \
+    adduser -u 1000 -G appuser -h /app -s /bin/sh -D appuser && \
+    chown -R appuser:appuser /app /logs /output
+
+USER appuser
+
+# Command to run
+ENTRYPOINT ["/app/traefik-log-processor"]
+CMD ["--config", "/app/config.yaml"]

README.md

@@ -1,106 +1,93 @@
-
 # Traefik Log Processor
 
-This Docker Compose setup uses the `hhftechnology/traefik-log-processor` image to process Traefik logs, splitting them into separate folders based on the "ServiceName" field and implementing log rotation and retention.
-
-## Introduction
-
-The `hhftechnology/traefik-log-processor` image contains a Python script that reads Traefik's JSON-formatted logs from standard input, parses each log entry, and writes it to a file in a directory structure based on the "ServiceName" and the date. It also performs periodic cleanup to remove old log files based on the retention policy.
-
-## Prerequisites
-
-- Docker
-- Docker Compose
-
-## Usage
-
-To use this log processor with your existing Traefik setup:
-
-1. **Configure Traefik to write logs in JSON format to a file.** For example, in your Traefik configuration:
-
-   ```yaml
-   [log]
-     filePath = "/logs/traefik.log"
-     format = "json"
-   ```
-
-   And in your Traefik Docker Compose service:
+A lightweight, resource-efficient tool that splits Traefik logs by service name while maintaining the original JSON format.
 
-   ```yaml
-   services:
-     traefik:
-       image: traefik:v3.3.4
-       volumes:
-         - traefik_logs:/logs
-       # ... other configurations ...
-   ```
+## Features
 
-2. **Create a Docker Compose file for the log processor:**
+- Splits Traefik JSON logs based on `ServiceName` field
+- Preserves original log format and structure
+- Supports multiple input methods (file, directory monitoring, stdin)
+- Configurable log rotation (size-based and time-based)
+- Configurable log retention policies (age-based and count-based)
+- Minimal resource footprint (written in Go)
+- Runs in a lightweight container
+- Simple configuration via YAML file
 
-   ```yaml
-   services:
-     log_processor:
-       image: hhftechnology/traefik-log-processor
-       volumes:
-         - traefik_logs:/input_logs
-         - processed_logs:/logs
-       command: tail -F /input_logs/traefik.log | python /app/process_logs.py
-   volumes:
-     traefik_logs:
-       external: true
-     processed_logs:
-   ```
+## Quick Start
 
-   This assumes that `traefik_logs` is the volume where Traefik writes its log file.
+```bash
+# Using Docker
+docker run -v /path/to/traefik/logs:/logs -v /path/to/output:/output \
+  -v /path/to/config.yaml:/app/config.yaml \
+  ghcr.io/hhftechnology/traefik-log-processor:latest
 
-3. **Start the log processor:**
-
-   Run the following command in the directory containing your `docker-compose.yml` file:
-
-   ```bash
-   docker compose up -d
-   ```
-
-The processed logs will be written to the `processed_logs` volume, organized by service name and date, e.g., `/logs/<service_name>/<YYYY-MM-DD>.log`.
-
-## Customization
-
-You can customize the log processing behavior using environment variables:
+# Using Docker Compose
+docker compose up -d
+```
 
-- `LOG_DIR`: Directory where processed logs are written (default: `/logs`)
-- `RETENTION_DAYS`: Number of days to retain logs (default: 30)
-- `CLEANUP_INTERVAL_HOURS`: Interval in hours between cleanup operations (default: 1)
+## Configuration
 
-For example, to change the retention period to 7 days, update your `docker-compose.yml`:
+Create a `config.yaml` file:
 
 ```yaml
-services:
-  log_processor:
-    image: hhftechnology/traefik-log-processor
-    environment:
-      - RETENTION_DAYS=7
-    volumes:
-      - traefik_logs:/input_logs
-      - processed_logs:/logs
-    command: tail -F /input_logs/traefik.log | python /app/process_logs.py
+input:
+  # Watch a single file
+  file: "/logs/traefik.log"
+  
+  # Or watch a directory for log files
+  # directory: "/logs"
+  # pattern: "*.log"
+  
+  # Or read from stdin
+  # stdin: true
+
+output:
+  # Base directory for service-specific logs
+  directory: "/output"
+  
+  # Format for service directories (supports templating)
+  format: "{{.ServiceName}}"
+
+rotation:
+  # Maximum size of each log file in MB
+  max_size: 100
+  
+  # Maximum age of each log file in days
+  max_age: 7
+  
+  # Maximum number of old log files to retain
+  max_backups: 5
+  
+  # Whether to compress old log files
+  compress: true
+
+# Optional field mapping (useful for CLF format or adding fields)
+field_mapping:
+  # You can rename fields or add new computed fields
+  # Example for adding a RouterName field:
+  # router_name: "{{extractRouterName .ServiceName}}"
 ```
 
-## Accessing Processed Logs
+## How It Works
 
-To access the processed logs from the host, you can mount the `processed_logs` volume to a host directory. For example:
+1. The application watches Traefik log files or reads from stdin
+2. Each log line is parsed as JSON
+3. The `ServiceName` field is extracted (e.g., `5-service@http`)
+4. The log entry is written to a service-specific directory
+5. Log rotation and retention policies are applied to manage storage
 
-```yaml
-volumes:
-  processed_logs:
-    driver: local
-    driver_opts:
-      type: none
-      o: bind
-      device: /host/path/to/processed/logs
+## Building from Source
+
+```bash
+git clone https://github.com/hhftechnology/traefik-log-processor.git
+cd traefik-log-processor
+go build -o traefik-log-processor cmd/main.go
 ```
 
-Then, the logs will be available at `/host/path/to/processed/logs/<service_name>/<YYYY-MM-DD>.log`.
+## Contributing
+
+Contributions are welcome! Please feel free to submit a Pull Request.
 
 ## License
 
-This project is licensed under the MIT License.
+This project is licensed under the MIT License - see the LICENSE file for details.