Blockers for ParticleOS remote builds + netboot via netesp

[bluca]

List of issues found when booting a netesp in vmspawn, and pointing to a uki-url that points to a particleos uki that sits next to a particleos ddi

• sd-repart does not run due to missing conditions:

  Condition: start condition unmet at Fri 2025-09-26 14:44:31 UTC; 4min 21s ago
             ├─ ConditionDirectoryNotEmpty=|/usr/lib/repart.d was not met
             ├─ ConditionDirectoryNotEmpty=|/usr/local/lib/repart.d was not met
             ├─ ConditionDirectoryNotEmpty=|/etc/repart.d was not met
             ├─ ConditionDirectoryNotEmpty=|/run/repart.d was not met
             ├─ ConditionDirectoryNotEmpty=|/sysusr/usr/lib/repart.d was not met
             └─ ConditionDirectoryNotEmpty=|/sysusr/usr/local/lib/repart.d was not met

as the DDI's /usr is not mounted in /sysusr despite having been downlaoded and attached:

● systemd-loop@run-machines-rootdisk.raw.service - Attach File /run/machines/rootdisk.raw to Loopback Block Device
     Loaded: loaded (/usr/lib/systemd/system/systemd-loop@.service; enabled-runtime; preset: disabled)
     Active: active (exited) since Fri 2025-09-26 14:44:38 UTC; 2min 32s ago

this is with rd.systemd.pull=raw,machine,verify=no,blockdev,bootorigin:rootdisk:ParticleOS__x86-64.raw in the kernel cmdline

• sd-repart does not work in the initrd when CopyBlocks is used repart: do not fail when CopyBlocks= is used in the initrd systemd#38700
• sd-repart does not work in the initrd when downloading an image, as by default it tries to operate on the initrd device repart: when running in the initrd use the ESP to find the root device systemd#38718

Aug 24 22:00:56 particle-132b-861e systemd[1]: Starting systemd-repart.service - Repartition Root Disk...
Aug 24 22:00:56 particle-132b-861e systemd-repart[397]: No machine ID set, using randomized partition UUIDs.
Aug 24 22:00:56 particle-132b-861e systemd-repart[397]: Can't fit requested partitions into available free space (0B), refusing.
Aug 24 22:00:56 particle-132b-861e systemd-repart[397]: Automatically determined minimal disk image size as 17.7G, current image size is 2.9G.
Aug 24 22:00:56 particle-132b-861e systemd[1]: systemd-repart.service: Main process exited, code=exited, status=1/FAILURE

needs to be pointed manually to the actual disk (ie: where the ESP is, should be able to find it automatically)

• sd-repart fails to CopyBlock from the usr-verity-sig partition just downloaded (usr and usr-verity work fine) Support CopyBlocks=auto for verity-sig partitions systemd#34835

root@particle-132b-861e:~# systemd-repart --dry-run=no /dev/vda
No machine ID set, using randomized partition UUIDs.
Unable to automatically discover suitable partition to copy blocks from for partition type usr-verity-sig.

• networkd configuration is not copied in the initrd, as mkosi.extra only goes to the final image mkosi-initrd: add 89-ethernet.network for network profile mkosi#3917
• network-manager does not work in the initrd preset: override networkd in desktop preset #98
• have to manually set a bunch of special magic sauce in the kernel command line, which means it cannot be used with secure boot or with a generic UKI

rd.systemd.pull=raw,machine,verify=no,blockdev,bootorigin:rootdisk:ParticleOS_x86-64.raw.zst

we should probably synthetize this from sd-boot if a uki-url entry is booted?
And the image name should be derived from the UKI filename + .raw + the default compression method, if not specified

• sd-importd fails when it runs twice, which is common when debugging in the emergency console in the initrd

Aug 24 22:09:28 particle-132b-861e systemd-importd[561]: (transfer1) Image 'rootdisk' already exists.
Aug 24 22:09:28 particle-132b-861e systemd-importd[561]: Transfer process failed with exit code 1.

• sd-importd and systemd-import@run-machines-rootdisk.raw.service sometimes fail to communicate and the latter fails, often when the network is slow and the latter times out. While it's receiving progress, it should probably avoid timing out?

Aug 24 18:58:38 particle-a19d-b14a systemd[1]: Starting systemd-import@run-machines-rootdisk.raw.service - Download of http://downloadcontentcdn.opensuse.org/repositories/system:/systemd/Debian_Testing_images/ParticleOS_x86-64.raw.zst...
Aug 24 18:59:23 particle-a19d-b14a varlinkctl[354]: Method call failed: Connection timed out

Aug 24 18:59:23 particle-a19d-b14a systemd-importd[355]: (transfer1) Got 35% of http://downloadcontentcdn.opensuse.org/repositories/system:/systemd/Debian_Testing_images/ParticleOS_x86-64.raw.zst. 1min 23s left at 3.5M/s.
Aug 24 18:59:23 particle-a19d-b14a systemd-importd[355]: Cannot emit log message varlink message, ignoring: Transport endpoint is not connected

• root=dissect and mount.usr=dissect do not work with the sd-import download hooks, after systemd-import@run-machines-rootdisk.raw.service completes there's no /dev/disk/by-designator/ that refers to the downloaded and opened DDI
• not sure if a repart or vmspawn issue, but booting the netesp with vmspawn --grow does not rewrite the GPT table, so repart still thinks there's no space and refuses to copy the partitions. Just running fdisk /dev/vda is enough to fix the GPT and make it recognize the full grown disk, and then repart can run. Not sure if repart should do this automatically, or vmspawn should do this when truncating a disk to new size, or both vmspawn: also adjust GPT headers with --grow systemd#38738

[poettering]

hmm, i am not following, why would sd-repart run in the initrd here? are you using this as an installer? i never tested that indeed.

[poettering]

• have to manually set a bunch of special magic sauce in the kernel command line, which means it cannot be used with secure boot or with a generic UKI

rd.systemd.pull=raw,machine,verify=no,blockdev,bootorigin:rootdisk:ParticleOS_x86-64.raw.zst

See documentation of "bootorigin". If you specify that and you are not actually booting via EFI network boot, then the whole request to download something is suppressed. I made sure of this to ensure the same EFI binary can be booted locally and remotely, and the right thing would happen.

[bluca]

hmm, i am not following, why would sd-repart run in the initrd here? are you using this as an installer? i never tested that indeed.

Well yes it needs to work as an installer, ie: you have the netesp locally, with enough space. You point sd-boot to the remote uki, which downloads the remote ddi, which has usr+verity+sig and that's it. So what happens next? You can't boot from the DDI, it has no space for anything else, and it's in memory only. The only disk is the netesp, and it has enough space, but only an ESP.

So repart needs to adjust the only disk available, and then copy the usr partition, and allocate the root/home/var partitions.

I'm getting a bit closer to that.

[bluca]

• have to manually set a bunch of special magic sauce in the kernel command line, which means it cannot be used with secure boot or with a generic UKI

rd.systemd.pull=raw,machine,verify=no,blockdev,bootorigin:rootdisk:ParticleOS_x86-64.raw.zst

See documentation of "bootorigin". If you specify that and you are not actually booting via EFI network boot, then the whole request to download something is suppressed. I made sure of this to ensure the same EFI binary can be booted locally and remotely, and the right thing would happen.

So you are saying that should always just be hardcoded in the UKI? But that takes a fair bit of space, and also you need to hardcode the image name which is not great.

It should at least have a mode to derive the image name from the uki name if not specified.

But then again, why not just synthetize a credential in the loader when loading a uki-url entry? That means zero configuration needed, which is nice

[poettering]

• have to manually set a bunch of special magic sauce in the kernel command line, which means it cannot be used with secure boot or with a generic UKI

rd.systemd.pull=raw,machine,verify=no,blockdev,bootorigin:rootdisk:ParticleOS_x86-64.raw.zst

See documentation of "bootorigin". If you specify that and you are not actually booting via EFI network boot, then the whole request to download something is suppressed. I made sure of this to ensure the same EFI binary can be booted locally and remotely, and the right thing would happen.

So you are saying that should always just be hardcoded in the UKI? But that takes a fair bit of space, and also you need to hardcode the image name which is not great.

well, nothing is free. I don't hink the expression is thaaaaaat long... so yeah, i'd just bake it in, and everything's clear.

It should at least have a mode to derive the image name from the uki name if not specified.

But then again, why not just synthetize a credential in the loader when loading a uki-url entry? That means zero configuration needed, which is nice

not sure i follow what you are suggesting

[bluca]

We do hit limits though, the cmdline isn't unlimited. The point is, if I boot an uki-url, and there's no disk, and there's a DDI where the UKI was, with the same name and version, and the auto-enrolled certs that were in the netesp verify it, that should be enough to automatically do the right thing, without having to manually add yet more imperscrutable magic strings that nobody knows about. Making things "just work" when they need to is important for a good end-to-end story