[CI] Add Terraform resources for daily CronJob that processes LLVM commits #495
Conversation
premerge/main.tf
Outdated
| } | ||
|
|
||
| # The container for scraping LLVM commits needs persistent storage | ||
| # for a local check-out of llvm/llvm-project |
There was a problem hiding this comment.
Why does this need to be stored persistently? It's pretty cheap to clone LLVM and a PVC I think adds unnecessary complexity on top of making things more complicated because they are now stateful.
There was a problem hiding this comment.
I neglected to mention this, but there's also a persistent file that keeps track of the last commits we've seen. Originally, the script was to run at a more frequent cadence so we wanted to keep track of commits we've seen as to avoid reprocessing them.
Now that the script only scrapes a day worth of data at a time, maybe we don't need a persistent state to keep track of commits we've seen. Although it might still be valuable for ensuring the quality of the commit data between iterations
There was a problem hiding this comment.
Using a PVC for a persistent file would make more sense.
I still think it's a bit of an antipattern though. If you want to ensure you're only looking at new commits and its a cron job, you can just look at the last 24 hours of commits (which it seems like you're already doing?). Making this stateless makes things quite a bit simpler and aligns things more with how k8s expects them to work.
There was a problem hiding this comment.
Addressed removal of dependency on persistent storage in #501
|
Please fix the terraform formatting before landing this though. At some point we might want to refactor this out into its own TF module, but for now I think this is reasonable enough. |
These resources are for a CronJob that executes the container at
ghcr.io/llvm/operations-metrics:lateston a daily basis (07:00 UTC), which will scrape daily metrics regarding LLVM's commit volume and upload them for visualization in Grafana.Changes were made to the already existing terraform files since many of the same resources are being reused anyway. This way we can keep all relevant changes in the same place instead of having two separate terraform directories that access and modify shared resources.
Since the container needs access to the BigQuery Google Cloud API, IAM and K8S service accounts were used to grant that access via Workload Identity Federation for GKE. More details at https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity