RFE: toolbox create --arch

[nwnk]

Is your feature request related to a problem? Please describe.

I have a powerful workstation of CPU architecture A, and I am developing for deployment on CPU architecture B.

Describe the solution you'd like
I would like to use toolbox to manage a container for architecture B, so that I can use a "native" development toolchain (because machine A is faster, or because I don't otherwise have access to an architecture B machine).

Describe alternatives you've considered

• Virtual machines are too heavyweight.
• Cross-compilers merely produce code, they don't run it.

Additional context

This came from an internal slack conversation:

<ajax> anyone know if it's possible to make toolbox create and manage a non-native-architecture container?
<nalind> doesn't appear to be expected, but it can be worked around, since it accepts the image and container names explicitly, and that image can be a reference to an image of a non-default arch
<nalind> for example the arm64 version of "fedora-toolbox:42", which including the registry part is "registry.fedoraproject.org/fedora-toolbox:42" skopeo inspect --raw docker://registry.fedoraproject.org/fedora-toolbox:42 lists an arm64 image with digest "sha256:1ae68d15077ea747834a20c834cf2617b058f3b452d417d37782d92287363dd9", and telling toolbox to use "registry.fedoraproject.org/fedora-toolbox@sha256:1ae68d15077ea747834a20c834cf2617b058f3b452d417d37782d92287363dd9", along with a manually-specified name for the container it bases on that image, worked over here
<ajax> hah, nice! that didn't work on the first try because "copy" grabbed the http-prefixed url link rather than just the image path, but:
<ajax> toolbox create -i registry.fedoraproject.org/fedora-toolbox@sha256:1ae68d15077ea747834a20c834cf2617b058f3b452d417d37782d92287363dd9 f42-arm
<ajax> some enterprising soul should hook that up to toolbox create --arch or something
<hadess> got an issue filed for that? i've used Flatpak runtimes for that in the past.

[nwnk]

Virtual machines are too heavyweight.

As a minor correction, I could see situations where the best way to manage this feature would be to use qemu-system-* to host a "passthrough" VM sort of like how podman machine works on macos. In particular, if the contained environment was sensitive to something like /proc/cpuinfo, it's going to be very surprised to find a Xeon when it's expecting ARMv9. So I'm not intrinsically opposed to a VM being involved here, but usually just one (per guest arch) if present.

[debarshiray]

I am shocked that this worked:

$ toolbox create --image registry.fedoraproject.org/fedora-toolbox@sha256:1ae68d15077ea747834a20c834cf2617b058f3b452d417d37782d92287363dd9 f42-arm

Toolbx's image handling code was never tested with SHA256 hashes, and I don't think anybody pushed it this far until now. :D

However, it soon runs into:

$ toolbox enter f42-arm 
Error: failed to bind /etc/machine-id to /run/host/etc/machine-id

... which is to be expected.

The container's entry point (ie., toolbox init-container) is a host architecture binary from the host, which manages to run inside the container because we inject the host's runtime environment (ie., the dynamic linker or PT_INTERP and all the shared libraries) into the container with some brutal hacks. It then shells out to the container's mount(8) binary to bind mount various things, which is of a different architecture and can't run on the machine.

How did you even manage to get the container to run? Running a container for an architecture that doesn't match the host machine will surely involve a virtual machine of some sort, no? The same machine can't run binaries for different architectures, can it? Or, am I missing something?

[debarshiray]

How did you even managed to get the container to run? Running a container for an architecture that doesn't match the host machine will surely involve a virtual machine of some sort, no? The same machine can't run binaries for different architectures, can it? Or, am I missing something?

Is this about different variants of the same CPU architecture (eg., variants of aarch64), as opposed to two entirely different CPU architectures (eg., aarch64 and x86_64)?

[nwnk]

It's primarily about different ISAs rather than different variants. It works (to the extent it works) because the host has qemu-user-static-aarch64 installed. But then:

Apparently you can fix this by mounting your own binfmt_misc within the child namespace and registering your own handler/flags. I haven't tried to make that work myself yet though. Obviously you wouldn't need that if a vm was being used, but that has its own issues (picking the kernel, for one).

[debarshiray]

It's primarily about different ISAs rather than different variants. It works (to the extent it works) because the host has qemu-user-static-aarch64 installed.

Okay, that makes sense now!

Yes, if we can get it to work somewhat smoothly out-of-the-box, then I am happy to add a --arch flag.