build-node-image: implement change detection and surface diff

[jlebon]

Right now, build-node-image will always build a new node and extensions container image. This is wasteful for anything watching the moving tag and thinking there's actually been a change. We should only push a new image if something actually did change.

Of course, change detection is notoriously hard when using a Dockerfile flow, unless e.g. your build is "semantically reproducible" and your git commit fully describes the contents (e.g. lockfiles). What we can do instead though is to just do the build, and then after the fact determine if something actually changed. If the git commit changed, then definitely push. If not, then do an RPM diff (of both the node and extensions images) and push if it changed.

As part of this we should add a FORCE flag to override change detection.

Also, since we have to calculate that information anyway, we might as well print the diff when we do have a change. Right now, diffs are only easily accessible if the node image ends up being part of a release payload (where the release controller can render that info), but we want visibility for every build.

[jlebon]

If the git commit changed, then definitely push. If not, then do an RPM diff (of both the node and extensions images) and push if it changed.

So the inputs that go into this build process are:

• OCP plashet
• RHEL repos (we do pull in a few RHEL packages as deps of the OCP packages)
• base RHCOS image (which itself is both RPMs and non-RPM content)
• specific files in openshift/os: primarily Containerfile, build-node-image.sh, and packages-openshift.yaml, scripts/generate-metadata, extensions/.

Presumably, if we have a "state digest" that combines all of those and includes that as a label on the image, we could have high-fidelity change detection.

In practice, it's probably not worth trying to achieve this. Instead, I think it would be good enough if we did:

• if the base RHCOS version changed, push
• if the git commit of openshift/os we built from changed, push
• if the RPM list changed, push (for this BTW, instead of downloading the whole image, we can do oc image extract "${image}[-1]" --file /usr/share/openshift/base/meta.json, and similarly for the extensions image).

[joelcapitao]

unless e.g. your build is "semantically reproducible" and your git commit fully describes the contents (e.g. lockfiles).

This can be done while onboarding openshift/os repo to Konflux. We have already the methodology and part of the code generating to lock files. We would need to create the Tekton task to achieve the release part (i.e pushing the content to the koji tag and openshift-release registry)

[joelcapitao]

if the base RHCOS version changed, push

The common way to achieve that in Konflux is by pinning to a specific digest the base image, and let MintMaker (i.e Renovate Konflux service) update it automatically when there is an update. The positive benefit is that it describes even more precisely the content in the git commit, hence a better reproducibility. But I don't know yet how this would impact current PROW configuration, and also the maintenance. This is something we need to investigate.

[jlebon]

Right yeah, I think if we can move the build node image job to Konflux soon, it's probably not worth trying to implement something fancier for this. Lockfiles + Konflux's path change triggering should be more than enough.

[jlebon]

To address my concern that "it would only lock in Konflux", it shouldn't be hard to adapt the build script slightly so that it reads in the Konflux lockfiles and feels that to treefile-apply (would require an override manifest or maybe we just move away from treefile-apply to a plain e.g. packages.txt file).