Skip to content

Offline verification makes network calls #4454

@apyrgio

Description

@apyrgio

Description

We have some CI jobs that perform lots of verifications using the cosign tool in offline mode. Our CI is in GitHub actions, which means that it's not airgapped, but we still want to test this code path.

What surprised us is that cosign verify-blob --offline can:

  1. Perform network calls to get some TUF certificates.
  2. Fail the operation, if the above calls fail.

We have been running this CI jobs for months now, and this issue has manifested once or twice. Here's a command that failed recently:

cosign verify-blob --offline --key /Users/runner/work/dangerzone/dangerzone/share/freedomofpress-dangerzone.pub --bundle var/folders/vk/nx37ffx50hv5djclhltc26vw0000gn/T/tmp5c90_qkr /var/folders/vk/nx37ffx50hv5djclhltc26vw0000gn/T/tmp75kgy45p

and its error:

Could not fetch trusted_root.json from the TUF repository. Continuing with individual targets. Error from TUF: error getting live trusted root: failed to create TUF client failed to load metadata: tuf refresh failed: Get "https://tuf-repo-cdn.sigstore.dev/13.root.json": dial tcp: lookup tuf-repo-cdn.sigstore.dev: no such host
Error: getting Rekor public keys: updating local metadata and targets: error updating to TUF remote mirror: tuf: failed to download 13.root.json: Get "https://tuf-repo-cdn.sigstore.dev/13.root.json": dial tcp: lookup tuf-repo-cdn.sigstore.dev: no such host
error during command execution: getting Rekor public keys: updating local metadata and targets: error updating to TUF remote mirror: tuf: failed to download 13.root.json: Get "https://tuf-repo-cdn.sigstore.dev/13.root.json": dial tcp: lookup tuf-repo-cdn.sigstore.dev: no such host

At the same time, we have verified that this command works in an airgapped mode, so it seems to me that there's a code path in cosign that does not ignore network errors, if it operates in offline mode, whereas other paths do

My questions are:

  1. Is the --offline mode a truly offline mode, that makes absolutely no network requests, or is it a mode where cosign relies on the signed timestamps, but may update its certificate chain from a CDN?
  2. Should cosign fail in a case like this?

Version

cosign version
  ______   ______        _______. __    _______ .__   __.
 /      | /  __  \      /       ||  |  /  _____||  \ |  |
|  ,----'|  |  |  |    |   (----`|  | |  |  __  |   \|  |
|  |     |  |  |  |     \   \    |  | |  | |_ | |  . `  |
|  `----.|  `--'  | .----)   |   |  | |  |__| | |  |\   |
 \______| \______/  |_______/    |__|  \______| |__| \__|
cosign: A tool for Container Signing, Verification and Storage in an OCI registry.

GitVersion:    v2.5.2
GitCommit:     af5a988bb15a03919ccaac7a2ddcad7a9d006f38
GitTreeState:  clean
BuildDate:     2025-06-17T20:03:14Z
GoVersion:     go1.24.4
Compiler:      gc
Platform:      linux/amd64

Note that the platform is not that important, since we've seen it fail both on macOS and Linux. Also, we purposefully use an older cosign version, to check how it behaves after a while (e.g. here, it triggered a certificate refresh).

Metadata

Metadata

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions