Discuss: maintaining and updating potentially large lists of CRD references in a library's config.jsonnet

[frimik]

Tooling and format for CRD references (index) maintenance

As mentioned in #516 (comment) I was thinking a bit about how certain libraries (operators, controllers) grow, it's no longer commonly one single CRD, there are sometimes a multitude. Personally the copy-pasting and potentially manual altering of the upstream list is tedious for those larger libs where it's not just as simple as copying a single CRD URL.

You could even miss adding a CRD if you're not carefully comparing the upstream old vs new CRD list. Some tooling may be suitable to aid.

Even though my example used vendoring, I don't think upstream "raw" vendoring, as in my extreme case, is a proper solution... it will get messy.
But some practices around local indexing formats may be something to think about... for example maybe a simple ./index/0.15.json for index keeping is a good practice?

local indexed_versions = [ // since 0.15
  { output: '0.15', version: '0.15.1', crds: (import './index/0.15.json').crds },
];

{
  "prefix": "^io\\.external-secrets\\..*",
  "crds": [
    "external-secrets.io_clusterexternalsecrets.yaml",
    "external-secrets.io_clusterpushsecrets.yaml",
    "...",
    "generators.external-secrets.io_stssessiontokens.yaml",
    "generators.external-secrets.io_webhooks.yaml"
  ]
}

I've been using various methods... curl, fully manual copy-pasting... kustomize localize to get a local copy of the upstream kustomize references for easily converting a directory list of yaml files into the resulting jsonnet list for example ...

@Duologic mentioned "central management" as a topic as well.. I have no quarrels so far with that. This is a great "service" 💯.
Any delays in getting MRs done is easily self-vendored as a workaround.

Best,
Mike

[frimik]

This code in combination writes a json index based on an upstream kustomization.yaml ...

kustomization-resources-to-index.sh:

#!/bin/bash

tag="$1"

if [[ -z "$tag" ]]; then
    echo "Usage: $0 <tag>"
    exit 1
fi

url="https://raw.githubusercontent.com/external-secrets/external-secrets/${tag}/config/crds/bases/kustomization.yaml"

KUSTOMIZATION=$(curl -sL "$url")

script_dir=$(dirname "$0")

jsonnet --tla-str kustomization="$KUSTOMIZATION" \
    ${script_dir}/kustomization-resources-to-index.jsonnet

kustomization-resources-to-index.jsonnet:

function(kustomization)
  local resources = std.parseYaml(kustomization)[0].resources;
  [
    resource
    for resource in resources
  ]

Such indexes could be used for example ...:

config.jsonnet:

local versions = [  // since 0.5
  { output: '0.5', version: '0.5.9', index: (import './index/0.5.9.json') },
  { output: '0.6', version: '0.6.1', index: (import './index/0.5.9.json') },
  { output: '0.7', version: '0.7.3', index: (import './index/0.7.3.json') },
  { output: '0.8', version: '0.8.12', index: (import './index/0.7.3.json') },
  { output: '0.9', version: '0.9.12', index: (import './index/0.7.3.json') },
  { output: '0.15', version: '0.15.1', index: (import './index/0.15.json') },
];

🤦‍♂ But, in this particular case (external-secrets) - while doing this, I even found out that the upstream kustomization.yaml file has been lying and was missing a CRD for a very long time. So I dunno.. all of my assumptions that you can find something reliable to write tooling against went out the window (for this repo at least historically, but maybe it's fine from now on). 😉

There is so much variation upstream that maybe my thoughts are just moot straight out the gate.

[Duologic]

There is so much variation upstream

Exactly this.

The central management isn't necessarily the problem, however it does put a strain on the maintainers of this repo. The community does a great job at bringing forward new versions, this currently needs to pass through a bottle neck for review. In this review, common issues arise like "shouldn't this be automated?" or "how many version should we support?".

My proposal would be to decentralise this a bit, make k8s-generator a GitHub Action that people can call on their own repo. Then how the updates are managed and how many version to keep becomes a problem distributed across the community.

The only drawback I see is a reduction in consistency and discoverability, perhaps we can run a small service to create an index of these libraries?

[frimik]

You mention version support, and old version support is always solveable by refs to branches or tags since the -libsonnet's are rendered into git repos! So it shouldn't be a strong reason that drives design I think. (Simple and standardized over complex and supporting "everything").

Maybe the old version issue can even be "pre-solved" by there always being a vX.Y lightweight tag matching the last existing version of a library. That way users could even pre-select their reference to a particular version that just always works, without having to lock to explicit git shas. Future surprise discovery that a library vanished from main while doing an update wouldn't even exist in this scenario. I haven't thought about automation for that yet but should be doable if the structure ("index"?) is standardized enough?

I don't know how idiomatic it is to actually use a "floating tag" for that... but something doesn't become idiomatic until somone starts doing it 😉. It is essentially also overkill, because you don't have to create this wonderful scenario for users. All that's needed is a note in a README 📖 that says DONT PANIC, and all old versions are always available by looking at old refs in git.

[xvzf]

Not been super active here lately except reviewing PRs, but adding my 2cents here:

I do think the generator could be beneficial as a "I run this as my own GitHub action / generate jsonnet-libs for my own library" and actually used it a similar at Ankorstore to generate jsonnet libs for an internal CRD.

However, I do see a big problem on discoverability (jsonnet-libs org has been quite good for this).

Wdyt about maintaining only libs for upstream Kubernetes and CNCF Graduated projects here? It'd ease the burden on maintainers & still offer a few ways for managing versions.

My proposal would be to decentralise this a bit, make k8s-generator a GitHub Action that people can call on their own repo. Then how the updates are managed and how many version to keep becomes a problem distributed across the community.

I do like this idea, but would simplify it further. This approach works when actual upstream maintainers of the software (e.g. ArgoCD/Flux) would manage their own jsonnet-libs generated by this action and offer it to the community. I don't see this happening for all projects as Jsonnet is still very niche.

I'd propose we offer a CLI entry point which generates a directory structure in the lib directory directly. The library would have to be committed to the repository where it's been used, but it would also allow an easy usage of the generator for internal CRDs as well.
A pattern I see for editors as well is fetching schema definitions from the live-Kubernetes context. Personally I don't prefer this but I do see a use-case for "generate me jsonnet-libs for the CRDs in my current cluster" as well.

[jemag]

As a relatively newcomer, I do agree that jsonnet-libs helps quite a bit in terms of discoverability and consistency.

Jsonnet is already fairly niche, without a ton of community articles/tutorials, and a lot of the existing resources are directing users towards jsonnet-libs.

While a GitHub action or CLI would be great additions, and I do like the idea of upstream maintainers (such as CNCF projects) maintaining their own library, if this is adopted I think there needs to be a clear mechanism to consistently discover those, preferably using jsonnet-libs as the source of that discovery.

If the plan is to simply decentralize and potentially for jsonnet-libs to disappear, I do think that would be a loss for new users and users considering adopting jsonnet. The jsonnet path is already a bit "harder" for users on the outside, and lowering discoverability/consistency would not be a step in easing that. The more new users we can bring in and keep into the community the healthier the Jsonnet ecosystem will remain over time, benefiting us all.

In the meantime, perhaps there are still some steps that can be taken to automate further and lower the maintenance of this repo. Also, if some upstream repos officially standardize their CRDs distribution, is there anything blocking us from 100% automating the lib update (e.g. self-merging PRs or at least automate the PR creation itself). Lastly, it might be worth considering adding maintainers. I very much appreciate all the amazing work that existing maintainers have been doing, nonetheless PRs can sometimes stay open for a decent chunk of time before seeing a review.