1.1.3 support: fork/exec permission denied

[agolomoodysaada]

When trying to install plugin on Vault 1.1.3, I receive this error

$ plugin_name=vault-plugin-secrets-kubernetes

$ plugin_sha256=$(kubectl exec -it -n vault "$VAULT_POD_NAME" -- sha256sum /vault/plugins/$plugin_name | awk '{print $1}')

$ vault write sys/plugins/catalog/secret/${plugin_name} sha256="${plugin_sha256}" command=${plugin_name}
Success! Data written to: sys/plugins/catalog/secret/vault-plugin-secrets-kubernetes

$ vault secrets enable -path=k8s -plugin-name=${plugin_name} plugin
Error enabling: Error making API request.

URL: POST http://localhost:8200/v1/sys/mounts/k8s
Code: 400. Errors:

* fork/exec /vault/plugins/vault-plugin-secrets-kubernetes: permission denied

[rvadim]

Hi. Sorry, Vault 1.x not supported yet. #6

[agolomoodysaada]

I was able to resolve this issue by applying chmod +x on the plugin binary.

However, I now get a different error

2019-08-07T23:24:04.758Z [ERROR] secrets.vault-plugin-secrets-kubernetes.vault-plugin-secrets-kubernetes_6cb633de.vault-plugin-secrets-kubernetes.vault-plugin-secrets-kubernetes: plugin init error: metadata=true error="open /tmp/plugin168290080: read-only file system" timestamp=2019-08-07T23:24:04.757Z
2019-08-07T23:24:04.855Z [ERROR] secrets.system.system_c5ff5c95: mount failed: path=k8s/ error="plugin exited before we could connect"

[rvadim]

Check that /tmp isn't mounted as a read only filesystem.

$ mount | grep /tmp

[agolomoodysaada]

Actually the / root path is mounted as read-only.
I can't figure out why?

/ # mount
overlay on / type overlay (ro,relatime,lowerdir=/var/lib/docker/overlay2/l/UZNRDXFFP3KD6FBMOAYFHFC2O6:/var/lib/docker/overlay2/l/DZ4DDDRES4C3Q24FPFQ7I2KBNN:/var/lib/docker/overlay2/l/H7SEBOQKWV7DBV63R4FG7U5B3O:/var/lib/docker/overlay2/l/XA6HS3XXP2F5ANXUU2GIBEVLLO:/var/lib/docker/overlay2/l/VRATD63NMG7ARHZJHOKGJIJOXL:/var/lib/docker/overlay2/l/T54OCS2V6ACUAQVRP3SM73JLXN,upperdir=/var/lib/docker/overlay2/bd203e6a522f3531b44d7577029e447b064b81b819925369142a9e474d230589/diff,workdir=/var/lib/docker/overlay2/bd203e6a522f3531b44d7577029e447b064b81b819925369142a9e474d230589/work)

[agolomoodysaada]

I figured out it was due to readOnlyRootFilesystem: true set on the security context on the official Helm chart. Worked around the problem by mounting /tmp as an emptyDir volume.
I'm still struggling to get the plugin working though...

Everytime I run

vault write k8s/config token="${SA_JWT_TOKEN}" api-url="https://$K8S_HOST:$K8S_PORT" CA=$(echo $SA_CA_CRT | base64)

I get

2019-08-12T14:53:25.604Z [INFO]  expiration: revoked lease: lease_id=sys/wrapping/wrap/h2212f2fbdcd8f62b30609543b7d3e673e8ac9a1459156369b41f8b57fc741dd1
2019-08-12T14:53:29.174Z [ERROR] secrets.vault-plugin-secrets-kubernetes.vault-plugin-secrets-kubernetes_4fca8cc9.vault-plugin-secrets-kubernetes.vault-plugin-secrets-kubernetes: plugin tls init: error="error during token unwrap request: Put https://vault-vault:8200/v1/sys/wrapping/unwrap: http: server gave HTTP response to HTTPS client" timestamp=2019-08-12T14:53:29.173Z
2019-08-12T14:53:29.273Z [ERROR] rollback: error rolling back: path=k8s/ error="plugin exited before we could connect"
2019-08-12T14:53:35.589Z [ERROR] secrets.vault-plugin-secrets-kubernetes.vault-plugin-secrets-kubernetes_4fca8cc9.vault-plugin-secrets-kubernetes.vault-plugin-secrets-kubernetes: plugin tls init: error="error during token unwrap request: Put https://vault-vault:8200/v1/sys/wrapping/unwrap: http: server gave HTTP response to HTTPS client" timestamp=2019-08-12T14:53:35.589Z
2019-08-12T14:53:35.673Z [ERROR] core: failed to run existence check: error="plugin exited before we could connect"

[bonifaido]

The Vault instance is configured without TLS. I suggest you to try the Banzai Cloud vault-operator to provision a Vault instance with TLS, it can even configure plugins for you automatically like this one.

https://github.com/banzaicloud/bank-vaults/blob/master/operator/README.md

[agolomoodysaada]

Do plugins not work without TLS? Is there a way to configure the protocol?

[agolomoodysaada]

After figuring out how to change the API url using VAULT_API_ADDR env var, I ran into this issue

2019-08-12T15:30:24.502Z [ERROR] secrets.vault-plugin-secrets-kubernetes.vault-plugin-secrets-kubernetes_4fca8cc9.vault-plugin-secrets-kubernetes.vault-plugin-secrets-kubernetes: plugin tls init: error="error during token unwrap request: Error making API request.

URL: PUT http://vault-vault:8200/v1/sys/wrapping/unwrap
Code: 403. Errors:

* permission denied" timestamp=2019-08-12T15:30:24.417Z
2019-08-12T15:30:24.513Z [ERROR] core: failed to run existence check: error="plugin exited before we could connect"

[bonifaido]

I would check the logs of Vault to see what has happened.

[agolomoodysaada]

Above are the vault logs