--podman-args passed arguments to subcommand instead of global as advertized

[zhangyi-hu]

Describe the bug

Args after --podman-args will be passed to subcommand (e.g. images) instead of global.

To Reproduce

podman-compose --podman-args="--connection remote-hostname" images

Expected behavior
Correct results on the images for containers on the remote server

Actual behavior
Errored out

Output

$ podman-compose --version 
podman-compose version 1.5.0
podman version 4.9.3

$ podman-compose --podman-args="--connection remote-hostname" images
...
raise subprocess.CalledProcessError(p.returncode, " ".join(cmd_ls), stderr_data)
subprocess.CalledProcessError: Command 'podman images --connection remote-hostname --format table my_container {{.Repository}} {{.Tag}} {{.ID}} {{.Size}} -n my_image:latest' returned non-zero exit status 125.

As you can see, the command generated is :
podman images --connection remote-hostname --format table my_container {{.Repository}} {{.Tag}} {{.ID}} {{.Size}} -n my_image:latest

The --connection remote-hostname is added after the images subcommand instead of right after podman.

In contrast, if I use the raw podman command with the correct order of the args, the results are as expected:

$ podman --connection remote-hostname images
REPOSITORY                 TAG         IMAGE ID      CREATED      SIZE
docker.io/library/busybox  latest      beae173ccac6  3 years ago  1.46 MB

[zhangyi-hu]

This is a blocking issue preventing our migration from docker-compose to podman-compose, as we rely heavily on remote api for service deployment.

[zhangyi-hu]

OK, After some investigation I find a way to work this around, so that podman-compose can work with remote host.

First of all, podman support remote host with cli arguments --connection followed by connection name created using podman system connection add. The bug with --podman-args makes this method useless for podman-compose. However, podman can also use environmental variables. What you need to do is to specify two environmental variables:

CONTAINER_HOST
CONTAINER_SSHKEY

example:
$export CONTAINER_HOST="ssh://<your-user>@<your_host>:<port>/run/user/<user_id>/podman/podman.sock"

The value of CONTAINER_HOST can be found by:
podman context ls or podman system connection ls under the column URI.

What is especially annoying is the fact that CONTAINER_SSHKEY or --identity needs to be specified explicitly, otherwise podman will not try any ssh key! A typical error message when your try to use remote host without specifying a ssh key contains the following phrase:

...Error: unable to connect to Podman socket: failed to connect: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none], no supported methods remain ...

OK, attempted methods [none], at least podman told us that it didn't try.

One more thing, the value of CONTAINER_SSHKEY has to be absolute path, it didn't expand ~.

From podman's official doc:

--remote, -r
When true, access to the Podman service is remote. Defaults to false. Settings can be modified in the containers.conf file. If the CONTAINER_HOST environment variable is set, the --remote option defaults to true.

This means that podman context use context_name is useless for podman-compose, because you need to set the CONTAINER_HOST anyway. The default label, that got assigned to connection_name after podman context use connection_name, can only be relevant when you use podman by podman -r or podman --remote without specifying a connection name, but that is impossible with podman-compose due to the bug reported in this issue.

To summarize:

The good news is you can use remote host with podman-compose.
The bad news is, that context switch like docker won't work, you need to manipulate your envars to achieve the same goal.

podman-compose can make the context switch work by simply offer a way to pass -r argument correctly to podman so that the user no longer need to play with the two envar mentioned above.