ci: add check on test coverage regression

[AndrewCharlesHay]

What would you like to be added:

Add a step in the GitHub Actions that checks the code coverage both before and after the PR. Ensure the new percent code coverage for each file is equal to or higher than the previous one.

Why is this needed:

Enforce test-driven development best practices.

[ivankatliarchuk]

I tried. Kubernetes org is quite burecratic kubernetes/org#5512

[TobyTheHutt]

I read the referenced issue. I wrote a quick abstract here. It's detailed because I thought it might be helpful for future reference. Maybe it also brings inspiration for a new approach towards solving this ticket.

For the impatient: Just read "Further Steps" at the end.

Need
A code-coverage reporting for the external-dns repo (and potentially others, too) to improve on test quality, early regression detection and to gain transparent metrics for contributors, users and gates.

Requirements

#	Title	Description
R01	Ownership	An app like Coveralls must be community owned and governed (no integrations under personal account governance).
R02	Audit	Ownership of the solution must be auditable.
R03	Per-Repo Permission Scopes	Scope of the solution to repos must be explicitly permitted for each appliance.

Constraints

#	Title	Description
C01	Org Access & CI Policy	Settings and access within the Kubernetes org are declaratively managed in kubernetes/org, reconciled by Peribolos and logged in the repo PR history.
C02	Change Restrictions	Changes to the org are proposed through a change PR to kubernetes/org and needs to be approved by an owner. If merged, Peribolos runs and applies the state.
C03	CI Governance	All CI automation jobs are defined as code in kubernetes/test-infra and Prow executes them as K8s pods via a ProwJob resource. To add or change a job or gate, a PR to kubernetes/test-infra adjusting job YAML must be reviewed and merged, so it gets deployed by Prow.

Solutions

#	Title	Description
S01	Prow-Native	Utilising jobs under kubernetes/test-infra to publish the coverage to dashboards or PR checks would remove the need for a new third-party app.
S02	Org-Installed App	An org owner installs an App like Coveralls at the organisation level and restricts it to target specific repos (service account and access will be recorded in Peribolos).
S03	Self-Hosted Coverage	We operate our own coverage service under CNCF/K8s control (which of course leads to high overhead and is overall atypical)

Conclusion
A quick benchmark of possible solutions makes it clear that S02 using Coveralls would require an org-owned and Peribolos documented exception.

S01 on the other hand does not conflict with any apparent rule or good practice within the organisation. In fact, the kubernetes/kubernetes repo already uses Prow-native coverage (e.g. the ci-kubernetes-coverage-unit job). Other repos also use the ci-kubernetes-coverage-unit and kubernetes-coverage-e2e-gci-gce Prow-jobs, as defined in the kubernetes/test-infra repo.

S03 is out of the question for several reasons, which is likely clear to anyone reading this. I only added it for completeness and reference.

Further steps
We might start to adopt the Prow-native checks (S01) as a safe and accepted baseline for our CI jobs. For this, we can open a PR in kubernetes/test-infra and add the wanted coverage jobs and PR check for externalDNS.

Further, if we still want Coveralls (S02), we will need to convince an org owner of its benefits for as, and install it scoped to our repo. This will require a new record for ownership, scope and rotation in kubernetes/org.

[ivankatliarchuk]

The AI generated report misses the point and please make sure to validate output. We currently have Prow enabled. But is nowhere near industry standards, available dashboards here https://github.com/kubernetes/test-infra. So prow does not support it.

Another option is to find something that does not require any 3rd party services or write own. This may get an approval from owners.

[ivankatliarchuk]

Related PR #5218

[AndrewCharlesHay]

I feel like I hit on something @ivankatliarchuk feels very strongly about 😅

[TobyTheHutt]

The AI generated report misses the point and please make sure to validate output. We currently have Prow enabled. But is nowhere near industry standards, available dashboards here https://github.com/kubernetes/test-infra. So prow does not support it.

This gnawed on me for a few days, tbh. 😅 No need to reply to this post - or even credit it, if you don't feel like it. Snowflake that I am, I just had to take position on that, so I can forget about it.

To varying extents, AI is present in many things I do. However, AI did not write my last post. The structure and content of chapters and tables is mine and it's how I work. Writing summaries like that belongs among the tools I regularly use and value.

If the post was unwelcome, or even annoying to some, I apologize. Apparently, my conclusions went into the wrong direction and I value the feedback regarding this.

[ivankatliarchuk]

My apologies, I didn't intend to cause offense or upset you. Your post came across as something an AI might have generated mainly due to it structure, but I was mistaken.