Command line argument values written as part of check_commandline to icingadb could contain credentials

[cisco-abrandel]

Describe the bug

Icingadb by default writes out the command, along with arguments used for a check, to the mysql database. This is visible in the check_commandline column of the service_state table.

I get the value in this, it allows you to see what was executed for a specific check and is great for debugging. The flip side of this is that its a very common pattern with nagios style checks to take credentials as command line arguments. Look no further than the ITL for lots of examples where checks take command line arguments that will contain credentials.

Being in the database is one thing, since access to this is likely limited, but the worse part of this issue is that icingaweb2 with the icingadb module will then show the check command and arguments in the source of a service. This means that anyone with access to view a service in icingaweb2 could potentially learn credentials, which is a pretty big issue.

While you could filter the display of these command line arguments in icingaweb, thats not the solution in my opinion. The database containing credentials in plaintext is the root of the issue here, and filtering the credentials in the UI is just a bandaid.

I understand that there is an argument to be made about never using command line arguments to pass credentials, but the reality is this is extremely prevalent, even in the ITL.

To Reproduce

Configure icinga2 with icingadb, and the icingaweb2 interface
Have some check that passes credentials as command line arguments

Expected behavior

Icingadb should provide the ability to redact command lines arguments from being written to the database, or a similar solution that doesn't involve credentials being exposed to users. I would imagine this be exposed as a knob that users can turn, since some might want the command line arguments exposed. Perhaps a more complete solution would be to allow users to redact the values of certain arguments, instead of just all of nothing.

Your Environment

• Icinga DB version: 1.1.1-g6c8b52f
• Icinga 2 version: 2.14.3
• icinga web version: v2.12.4
• Operating System and version: Rocky Linux 9

[oxzi]

Thanks for your report! This issue did not appear with Icinga DB, but was already a potential headache in Icinga 2 using the IDO. IIRC there were some related issues, e.g., Icinga/icinga2#6404.

While you have mentioned restrictions in Icinga DB Web, I just wanted to point out that it is possible to restrict access to an object's source or to certain variables¹. However, one could argue that it is better not to collect sensitive data at all. Speaking of which, I would personally discourage using command line arguments for sensitive information at all, as they are visible to all other system users by default².

IMO, your request is valid and necessary in certain setups. Instead of "encrypting" arguments or custom vars, redacting or filtering is something we could evaluate in Icinga DB.

Footnotes

1. https://icinga.com/docs/icinga-db-web/latest/doc/04-Security/ ↩

2. This advise is not very helpful, I know. ↩

[julianbrost]

IMO, your request is valid and necessary in certain setups. Instead of "encrypting" arguments or custom vars, redacting or filtering is something we could evaluate in Icinga DB.

In Icinga DB? How? If anything, this has to be done in Icinga 2, otherwise you can simply repeat the same argument for Redis (where parts of the source tab are fetched from directly).

[cisco-abrandel]

Thanks for the feedback! I agree there are ways to mitigate what I highlight here, the number one being just to not pass credentials on the command line. This might be a tough sell for some people given how prevalent this is, but I don't disagree at all, if you have access to the box where the checks are running its trivial to see the arguments passed to any command.

I still feel that writing out command line arguments to data stores in an ecosystem where command line arguments commonly contain sensitive information isn't ideal. It makes these data stores a juicy target for attackers, and in my mind its not obvious that this information ends up there.

[slalomsk8er]

Yes, environment variables are invisible to ps but still not ideal.

AFAIK, state-of-the-art would be, some kind of second channel, maybe via icinga2 API, to fetch credentials from a "vault" where the accessible secrets could be limited and cached in the cluster, depending on the involved objects (check, host and service) and the command_endpoint, combined with some in-RAM secret hardening.

This would probably require coordination with the check plugin developers, so involving the big collection maintainers like monitoring-plugins.org, labs.consol.de, and Linuxfabrik (@markuslf) would be in order, to define a standard interface.

[markuslf]

Monitoring plugins follow the Nagios standard and are normally agnostic to the monitoring software used.

The source of the problem is monitoring plugins that require credentials on the command line (as some of our plugins still do). This is where we need to start. Plugins that are actively under development could be extended to authenticate via (hidden) credential files first, as is the case with our plugins for mysql/mariadb, for example. This is the most simple solution. Other sources such as real vaults like Hashicorp, Ansible etc. could also be used, but this is probably not realistic due to complexity and other pitfalls.

While not providing credentials on the command line is the best approach, obfuscation could be used to clean up sensitive data. For example, Icinga2 as a central repository for command line calls could be configured to hide credentials directly. However, other tools such as Graylog or the ELK stack also face the same problem. As an inspiration, Graylog allows the use of pipelines to help remove the data from the message or change the value of the message to a hash value.

[slalomsk8er]

@markuslf The problem with files is that they need to be deployed somehow. I know that it works for you but not everyone can just add the secret into an host file in the Ansible inventory. But even then, do the files get removed if no longer needed or do credentials stay behind as forgotten remnants?

To keep Monitoring plugins agnostic and compatible to the Nagios standard it could be possible to just try different methods from most secure to least secure if no credentials are given but required or add an optional argument to indicate the method to be used.

Anyway, it would be great if this would also solve the problem of secrets being accidentally written to files on every host with an Icinga2 agent. This was my reasoning to mention a vault that allows resolving and caching secrets by involved objects (check, host and service) and the command_endpoint. Every node is authenticated but is it also authorized to access the secret?

I would propose a mechanism to optionally allow a check plugin to talk to Icinga2 to acquire the needed credentials. Maybe stdin or a socket could also work? For stdin the check object would need to trigger the credential retrieval to have it ready to hand it over.

With obfuscation via pipeline, do you mean something like a plug-gable function to the Icinga2 features that write data out to DBs and such?

https://community.icinga.com/t/rfc-a-new-monitoring-plugins-interface/13880/4