Nomos vet fails to validate OPA Gatekeeper custom resource when ConstraintTemplate is in the same commit

[philipsabri]

When validating a set of Kubernetes configuration files using nomos vet, specifically involving OPA Gatekeeper resources, a validation failure occurs if a ConstraintTemplate (which defines a Custom Resource Definition) and a corresponding Constraint (an instance of the custom resource defined by the template) are present together in the same batch of files being validated (e.g., as part of a single commit or PR).

This behavior seems specific to Gatekeeper ConstraintTemplates. nomos vet does not exhibit the same failure when validating a standard CustomResourceDefinition object alongside an instance of its defined kind present in the same batch of files. It is likely that similar issues would occur with other tools that generate CRDs indirectly, such as kro.

This forces users to either split the ConstraintTemplate definition and its corresponding Constraint usage into separate commits/validation steps (which hinders atomic changes).

Steps to reproduce

1. Install OPA gatekeeper
2. run nomos vet on this example

kind: ConstraintTemplate
apiVersion: templates.gatekeeper.sh/v1
metadata:
  name: validateuserinfo
spec:
  crd:
    spec:
      names:
        kind: ValidateUserInfo
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8svalidateuserinfo
        violation[{"msg": msg}] {
          username := input.review.userInfo.username
          not startswith(username, "system:")
          msg := sprintf("username is not allowed to perform this operation: %v", [username])
        }
---
kind: ValidateUserInfo
apiVersion: constraints.gatekeeper.sh/v1
metadata:
  name: always-validate

3. Error

Error: KNV1021: No CustomResourceDefinition is defined for the type "ValidateUserInfo.constraints.gatekeeper.sh" in the cluster.
Resource types that are not native Kubernetes objects must have a CustomResourceDefinition.

Config Sync will retry until a CustomResourceDefinition is defined for the type "ValidateUserInfo.constraints.gatekeeper.sh" in the cluster.

source: /workspaces/kpt-config-sync/philip/cluster/test.yaml
metadata.name: always-validate
group: constraints.gatekeeper.sh
version: v1
kind: ValidateUserInfo

For more information, see https://g.co/cloud/acm-errors#knv1021

Any ideas or thoughts on how we could unblock this? Maybe we could skip specific CRD API groups, or handle ConstraintTemplate similarly to how we handle CustomResourceDefinition. I would appreciate a workaround or fix as GKE "advertises" using Gatekeeper ConstraintTemplates and enabling Gatekeeper on a cluster with Policy Controller.

[sdowell]

It looks like the core issue is that Gatekeeper maps a ConstraintTemplate to a CustomResourceDefinition, but nomos vet doesn't know about ConstraintTemplate or how to map it to a CustomResourceDefinition.

Since nomos vet doesn't know about the CustomResourceDefinition, it tries to look up the CRD in the cluster and fails to find it since the CRD does not exist yet on the cluster. There are a few workarounds that come to mind:

• You could use Gatekeeper to generate the CRD on a test cluster and then cache the CRD locally before running nomos vet
• You could pass the --no-api-server-check flag to nomos vet. This will skip validation for unknown resource kinds (e.g. Custom Resources)

[philipsabri]

Thanks for the reply!

• You could pass the --no-api-server-check flag to nomos vet. This will skip validation for unknown resource kinds (e.g. Custom Resources)

I was thinking about this, but I would only want to use --no-api-server-check for specific API groups, not for all unknown resource kinds.

We somewhat achieved this by removing all the affected resource files with the specific API group as a step in our CI pipeline before running nomos.

[sdowell]

I was thinking about this, but I would only want to use --no-api-server-check for specific API groups, not for all unknown resource kinds.

I think this would be a reasonable feature request

[samirtahir91]

@philipsabri I have similar situation in my org specifically for gatekeeper, our workaround is separating all gatekeeper resources into a global policy repo with nomos vet using --no-api-server-check, but agree this change would be beneficial for future use cases/specific crds