Skip to content

Conversation

Copilot
Copy link

@Copilot Copilot AI commented Sep 25, 2025

Problem

OCI artifacts generated by the proxy had unstable checksums, causing FluxCD to detect "new" artifacts on every reconciliation cycle even when the chart content was unchanged. This resulted in unnecessary helm upgrade operations and continuous reconciliation events like:

Normal  ArtifactUpToDate  artifact up-to-date with remote revision: '1.13.3@sha256:01670a198f036bb7b4806c70f28c81097a1c1ae993e6e7e9668ceea3c9800d69'
Normal  ArtifactUpToDate  artifact up-to-date with remote revision: '1.13.3@sha256:2565063055a68c060dcd8754f5395bc48ebcf974799a9647d077f644bf29a584'

The same chart version (1.13.3) was producing different checksums each time it was processed.

Root Cause

In internal/manifest/dst.go, the CreatedAt field was set to time.Now() each time a manifest was created, causing identical chart content to have different timestamps and therefore different checksums.

Solution

Implemented deterministic timestamp generation that ensures reproducible OCI artifacts:

  1. Primary: Use the chart's Created timestamp from the Helm repository index when available
  2. Fallback: Generate a deterministic timestamp using SHA256 hash of chartName@version
  3. Compatibility: Fall back to time.Now() only when no chart version information is available

Changes Made

  • Modified internal/manifest/dst.go:

    • Added chartVer field to InternalDst struct to store chart metadata
    • Added NewInternalDstWithChartVer() constructor
    • Implemented getDeterministicTimestamp() method
    • Updated Push() method to use deterministic timestamps
  • Updated internal/manifest/charts.go:

    • Modified prepareChart() to pass chart version data to the destination

Testing

Added comprehensive tests covering:

  • Basic reproducibility verification
  • Ingress-nginx chart scenario (as requested in issue)
  • FluxCD reconciliation simulation showing identical checksums across multiple cycles
  • Fallback behavior when no Created timestamp is available

Result

The same chart version now produces identical checksums every time:

  • ✅ FluxCD no longer detects false "new" artifacts
  • ✅ Eliminates unnecessary helm upgrades
  • ✅ Maintains backward compatibility
  • ✅ Works with all existing chart repositories

Fixes #[issue-number] where FluxCD was continuously reconciling due to unstable OCI artifact checksums.

Original prompt

This section details on the original issue you should resolve

<issue_title>The resulting OCI artifact does not have a stable checksum.</issue_title>
<issue_description>The The resulting OCI artifact that gets downloaded does not have a stable checksum.

In cionjunction with FluxCD, this results on a "new" detected artifact, each time the OCIRepository gets reconciled:

e.g. with hetstack/cert-manager:

  Normal  ArtifactUpToDate            47h                    source-controller  artifact up-to-date with remote revision: '1.13.3@sha256:01670a198f036bb7b4806c70f28c81097a1c1ae993e6e7e9668ceea3c9800d69'
  Normal  ArtifactUpToDate            43h                    source-controller  artifact up-to-date with remote revision: '1.13.3@sha256:2565063055a68c060dcd8754f5395bc48ebcf974799a9647d077f644bf29a584'
  Normal  ArtifactUpToDate            41h                    source-controller  artifact up-to-date with remote revision: '1.13.3@sha256:e3d7f4eb00c95d556c372f2b8a3a7c8a8c28ba019f3e10928e118963d083e786'
  Normal  ArtifactUpToDate            38h                    source-controller  artifact up-to-date with remote revision: '1.13.3@sha256:b8d6f76c8c90b5c99eb0cfd7eb21bc5368c9006c5f882121d647c5cbddcc8203'
  Normal  ArtifactUpToDate            36h                    source-controller  artifact up-to-date with remote revision: '1.13.3@sha256:d7d92b499afeb5d8679fc68149299a127c181f516d732496d4a3a674a17109a7'
  Normal  ArtifactUpToDate            34h                    source-controller  artifact up-to-date with remote revision: '1.13.3@sha256:8e2675a792a92485e33efec225d489ae092e30bc03e02840c7e14514323253c2'

Which in turn triggers a helm upgrade each time.</issue_description>

<agent_instructions>make sure, when you download the tgz file and convert it to OCI that the artifact
generated is reproducible. generating for the same tgz input the same output with matching hash. Create a test case to verify that with helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
</agent_instructions>

Comments on the Issue (you are @copilot in this section)

@Vad1mo I see this is indeed a bit of a problem..
Fixes #32

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.


Summary by cubic

Make OCI artifact generation reproducible for a given Helm chart tgz, producing a stable checksum and stopping FluxCD from triggering upgrades on each reconcile. Fixes #32.

  • Bug Fixes
    • Make tgz-to-OCI conversion deterministic (consistent layer ordering and metadata).
    • Emit the same digest for identical input across runs.
    • Add a reproducibility test using the ingress-nginx chart that converts the same tgz twice and asserts matching digests.

Copilot AI and others added 3 commits September 25, 2025 15:46
Co-authored-by: Vad1mo <1492007+Vad1mo@users.noreply.github.com>
Co-authored-by: Vad1mo <1492007+Vad1mo@users.noreply.github.com>
Co-authored-by: Vad1mo <1492007+Vad1mo@users.noreply.github.com>
@Copilot Copilot AI changed the title [WIP] The resulting OCI artifact does not have a stable checksum. Fix OCI artifact checksum instability causing FluxCD reconciliation issues Sep 25, 2025
@Copilot Copilot AI requested a review from Vad1mo September 25, 2025 16:03
Copilot finished work on behalf of Vad1mo September 25, 2025 16:03
@Vad1mo Vad1mo added the bug Something isn't working label Sep 26, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The resulting OCI artifact does not have a stable checksum.

2 participants