Added Prometheus, Grafana and starting dashboards for Minio and Flink

[gordonmurray]

No description provided.

[Copilot]

Pull Request Overview

This PR adds comprehensive monitoring capabilities by integrating Prometheus for metrics collection and Grafana for visualization. It includes pre-configured dashboards for monitoring MinIO and Flink components in the data processing pipeline.

• Adds Prometheus and Grafana services with proper service discovery configuration
• Creates monitoring dashboards for MinIO storage metrics and Flink job/task manager performance
• Configures metrics exporters for both MinIO and Flink with appropriate endpoint exposure

Reviewed Changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
monitoring/prometheus.yml	Prometheus configuration with scrape targets for MinIO and Flink components
monitoring/grafana/provisioning/datasources/prometheus.yml	Grafana datasource configuration pointing to Prometheus
monitoring/grafana/provisioning/dashboards/minio-dashboard.json	MinIO monitoring dashboard with storage and request metrics
monitoring/grafana/provisioning/dashboards/flink-dashboard.json	Flink monitoring dashboard with CPU and memory usage panels
monitoring/grafana/provisioning/dashboards/dashboards.yml	Dashboard provisioning configuration for Grafana
docker-compose.yml	Adds Prometheus and Grafana services with volume mounts and networking
conf/flink-conf.yaml	Enables Prometheus metrics reporter for Flink components
Dockerfile	Ensures proper permissions for Flink Prometheus plugin directory

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}