use case: visualise metrics stored as json files in S3

[AmitKumarDas]

ProblemStatement: As an automation engineer I collect various metrics as json format & upload them to S3. I want to visualise the same on Grafana like visualisation tools running in some K8s setup.

[AmitKumarDas]

High Level Design Drafts - 0

This design is split across multiple custom resources:

- [1] Fetch metrics in json files from S3

• Command resource should delete resources that were created by it earlier
• Command custom resource to fetch files from S3
• This Command resource should run every day once (configurable)
• A TTL based ConfigMap can act as a lock to reconcile Command after 24 hours only
• Command resource should wrap each metrics file as a ConfigMap with labels
• Any failure in reconciling Command resource should be retried in some interval
• Command status should say when the last successful reconciliation was done
• Command resource will apply a new custom resource called PrometheusLoader

- [2] Upload json files containing metrics to Prometheus

• Implement PrometheusLoader as a custom resource
• It should translate all the relevant ConfigMaps as Prometheus metrics & upload the later
• Schema to translate data from json structure to prometheus metrics needs to be designed
• This schema itself will be a json structure that can be wrapped inside a ConfigMap
• Any failure to reconcile PrometheusLoader will be retried

- [3] Implement a TTL based Lock

• Existence of a ConfigMap is considered as a lock
• This ConfigMap can be created by other controllers with exact labels
• This ConfigMap should also have a label with Time To Live i.e. TTL
• The corresponding controller will delete this ConfigMap once its TTL expires

NOTE:
- All the custom resources will be implemented as ConfigMaps
- This will be done to avoid the need of applying Custom Resource Definition(s)

[AmitKumarDas]

refer - https://github.com/prometheus-community/json_exporter
refer - https://github.com/NexClipper/exporterhub.io