feat: use content fetch instead of image pull

[awesomenix]

What this PR does / why we need it:

We should just fetch the contents instead of performing a pull, so the compressed images remain on disk, when a run is performed all it does is unpack and run. The difference is negligible at cost of smaller initial image size.

here is the difference

Before

21G	/var/lib/containerd

After

6.0G	/var/lib/containerd

Which issue(s) this PR fixes:

Fixes #

Requirements:

• uses conventional commit messages
• includes documentation
• adds unit tests
• tested upgrade from previous version
• commits are GPG signed and Github marks them as verified

Special notes for your reviewer:

Release note:

none

[djsly]

if this work, it will be a game changer:

ctr images pull
What it does

Resolves the image reference (tag/digest)
Fetches:

Image manifest
Config object
All layer blobs


Registers the image in containerd’s image store
Makes it available to:

ctr run
nerdctl run
Kubernetes (via CRI)

ctr content fetch
What it does

Fetches content by descriptor or ref
Stores blobs only in the content store
Does not:

Create an image
Register it for execution
Resolve dependencies automatically (unless explicitly referenced)

Why would anyone use ctr content fetch?

2. Air‑gapped or mirrored registries

Pre-fetch blobs
Assemble images later
Control exact digests

[djsly]

ctr content fetch <ref|descriptor>

Downloads raw OCI blobs (manifest, config, layer tarballs) into the content store.
Does not create an image record in the image metadata store.
Does not unpack layers into the snapshotter (unless you later do it).
Result: you have the building blocks locally, but no runnable image is registered.

ctr images pull

Resolves the reference, downloads the same blobs into the content store, and creates the image record.
Optionally unpacks layers (--unpack) to the snapshotter for faster first start.
Result: runnable image appears in ctr images ls and is usable by ctr run, CRI, etc.

[djsly]

1) Fetch (blobs land in content store)

ctr content fetch docker.io/library/nginx:latest

2) Identify the manifest digest you fetched

ctr content ls | grep manifest

e.g. sha256:MANIFEST_DIGEST

3) Create the image record (register name → manifest)

ctr images create
docker.io/library/nginx:latest
sha256:MANIFEST_DIGEST

4) (Optional) Unpack now to avoid first-run penalty

ctr images unpack docker.io/library/nginx:latest

5) Run

ctr run --rm docker.io/library/nginx:latest nginx

If you skip step 3, the image won’t show up in ctr images ls, and ctr run won’t find it.
If you skip step 4, the first run will pay the unpack cost.

[awesomenix]

Having a penalty overhead of first time run is expected, but the tradeoff here is caching many more images since we have disk space of 15GB+ vs smaller set right now with unpacked.

Did a lot of testing, there is an overall penalty of 5s-7s additional startup time, to unpack and register the image with snapshotter (overlayfs). But this penalty is across all the containers in parallel.

with content fetch

ctr content fetch mcr.microsoft.com/aks/aks-gpu-cuda:580.95.05-20251021155213
mcr.microsoft.com/aks/aks-gpu-cuda:580.95.05-20251021155213: resolved |++++++++++++++++++++++++++++++++++++++|
index-sha256:0030c917ff77ddcf9178b308183f3206e87a09caa469919091c56f636b2f085d: done |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:ceb88ac8edb5d5784e67f3ab2ecdf5326bbe17c5d358b3113b155f3864c4b48d: done |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:12fe821b53d7d08fa94921798357863e98d370b8d4b1b84901710630e406176e: done |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:35e54a72f0aaddd65cf5651ed7ee3b5ce7ae8d6e988c4e56b4db2899013ab7ed: done |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:7abab46f17154bd8a84006f1d346930e52dc7eaa1369e92f2871d8e1a68d29f9: done |++++++++++++++++++++++++++++++++++++++|
config-sha256:18b8431d08c9cca89bdd05716d3bf66d2766fa6bf47b85fd0129aad25bffdb5a: done |++++++++++++++++++++++++++++++++++++++|
config-sha256:6081dea4d21051c04e278b83406025041126e67fff28f8f0fb9283bbefcd89e0: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:4b3ffd8ccb5201a0fc03585952effb4ed2d1ea5e704d2e7330212fb8b16c86a3: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:db710fc699676318869ea835688cf75c00aa98f3ac2d252de9d87ce40f528fc5: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:9ee85a502d0b32e2a344216a072a7f0462e752efb373348773201da82665e389: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:8f60597c1f34e69884127c7177870f0c93b47c4be8ed44574d00a40a510f35b8: done |++++++++++++++++++++++++++++++++++++++|
config-sha256:0e8e2e177580d419b05b8ce672464ba949ed8203db09689fd4712b0145b7429d: done |++++++++++++++++++++++++++++++++++++++|
config-sha256:e76791a520b21300630a3e4b891203f07c09a6f5243d12c7dd6f74c673c98877: done |++++++++++++++++++++++++++++++++++++++|
elapsed: 8.0 s total: 676.5 (84.6 MiB/s)

time ctr containers create mcr.microsoft.com/aks/aks-gpu-cuda:580.95.05-20251021155213 debug-box

real 0m7.905s
user 0m0.019s
sys 0m0.025s

with images pull

time ctr images pull mcr.microsoft.com/aks/aks-gpu-cuda:580.95.05-20251021155213
mcr.microsoft.com/aks/aks gpu cuda:580.9 saved
└──index (0030c917ff77) complete |++++++++++++++++++++++++++++++++++++++|
├──manifest (35e54a72f0aa) complete |++++++++++++++++++++++++++++++++++++++|
│ ├──config (6081dea4d210) complete |++++++++++++++++++++++++++++++++++++++|
│ ├──layer (4f4fb700ef54) complete |++++++++++++++++++++++++++++++++++++++|
│ ├──layer (4b3ffd8ccb52) complete |++++++++++++++++++++++++++++++++++++++|
│ ├──layer (9ee85a502d0b) complete |++++++++++++++++++++++++++++++++++++++|
│ ├──layer (db710fc69967) complete |++++++++++++++++++++++++++++++++++++++|
│ └──layer (8f60597c1f34) complete |++++++++++++++++++++++++++++++++++++++|
├──manifest (ceb88ac8edb5) complete |++++++++++++++++++++++++++++++++++++++|
│ └──config (e76791a520b2) complete |++++++++++++++++++++++++++++++++++++++|
├──manifest (12fe821b53d7) complete |++++++++++++++++++++++++++++++++++++++|
│ └──config (18b8431d08c9) complete |++++++++++++++++++++++++++++++++++++++|
└──manifest (7abab46f1715) complete |++++++++++++++++++++++++++++++++++++++|
└──config (0e8e2e177580) complete |++++++++++++++++++++++++++++++++++++++|
application/vnd.oci.image.index.v1+json sha256:0030c917ff77ddcf9178b308183f3206e87a09caa469919091c56f636b2f085d
Pulling from OCI Registry (mcr.microsoft.com/aks/aks-gpu-cuda:580.95.05-20251021155213) elapsed: 16.4s total: 676.5 (41.4 MiB/s)

real 0m16.372s
user 0m0.010s
sys 0m0.018s

time ctr containers create mcr.microsoft.com/aks/aks-gpu-cuda:580.95.05-20251021155213 debug-box

real 0m0.054s
user 0m0.015s
sys 0m0.012s

[awesomenix]

Obviously we can use this option strategically, like use images pull for critical containers which run by default on first time bootup, like pause, coredns etc.

i feel that 5s-7s additional startup time is not much, compared to savings we get in additional areas like disk space, additional caching etc.

But open to discuss as a team on pros and cons.