openrc-run: default supervisor stdout and stderr to logger

[navi-desu]

Currently we redirect any stdout and stderr output to /dev/null. This is making debugging, or helping others debug, unnecessarily harder, so let's just assign a sane default that should work on most systems.

[N-R-K]

I'm not sure if this is a good idea. User can just enable logging if they need to? But to underscore my point, "sane default" is subjective, and I think /dev/null is the sane default as it doesn't assume that the user wants to log unless told to.

Adding rc_{output,error}_logger may be useful. But IMO they should be unset by default.

Also manpages describe that "Cmd must be an absolute pathname". So the current default in this PR may not work with chroot.

[navi-desu]

systemd and s6 both have catch-all loggers, though to journald and s6-log respectivelly

and that immensely helps catching issues, specially when you might not know what happened in a server until it's way after, or when trying to help people debug intermittent issues

differently than systemd and s6 tho, we don't ship a logger-- but posix logger is standard and given openrc's nature, fits as our default logger

text doesn't use a lot of space, default configs for syslogds now a days have logrotate enabled by default, and most well behaving daemons don't spew a lot of trash in std{out,err} anyway- the use case for this is exactly the cases where people won't think of setting up a logger, to later figure out they should've

[N-R-K]

we don't ship a logger

This is an important point because it means that unlike systemd, openrc users need to manually install and configure a logger to begin with. And so setting a value to rc_{output,error}_logger also is not going to be a huge deal.

but posix logger is standard

The posix logger defines no option (or anything else really) so the options used in this PR aren't exactly "posix".

[navi-desu]

we don't ship a logger

This is an important point because it means that unlike systemd, openrc users need to manually install and configure a logger to begin with. And so setting a value to rc_{output,error}_logger also is not going to be a huge deal.

logger is required by posix, and thus we can assume it'll be available in a posix system, and nothing really breaks if there's no syslogd set up (posix defines the logger tool to be implementation, it might not even use a syslogd depending on the platform)

setting rc_*_logger is not a big deal if people are aware of it, getting aware of it after things go wrong and there's no log about it is the not great is an awful experience, seeing a few log lines and wanting to turn that off is a better one

but posix logger is standard

The posix logger defines no option (or anything else really) so the options used in this PR aren't exactly "posix".

https://pubs.opengroup.org/onlinepubs/9799919799/utilities/logger.html

posix logger does define every option i've used here since posix.2024, so it is posix (except for the daemon facility, but esyslog in libeinfo already uses it, so it's on every platform we support)

[N-R-K]

https://pubs.opengroup.org/onlinepubs/9799919799/utilities/logger.html

posix logger does define every option i've used here since posix.2024

Okay, I see. My local manpages didn't include it.

getting aware of it after things go wrong and there's no log about it is the not great is an awful experience

There's 3 scenarios here.

1. Default case of user not configuring any logging.

And I don't mean the logger binary here. I have it installed on my system from util-linux package which enables +logger by default. But it doesn't log anything since I have no logging facility set up (as far as I can tell, it fails to open "/dev/log" which doesn't exist and then just proceeds to basically just exist yet do nothing useful for the rest of it's lifetime).

In this case, enabling logging by default doesn't help in the situation you're describing.

2. Logging configured, logger manually set for services/openrc

Also doesn't help.

3. Logging configured, logger not set for services/openrc

This is the only situation where setting it helps.

---

Okay so how about this: if rc_{output,error}_logger is not set by the user, then instead of statically setting it to logger ... it instead probes if logging is actually active or not and sets a default if it is otherwise doesn't set anything.

It will help in situation 3 while also not spawning a bunch of useless logger process in situation 1.

[nmeum]

Just to throw this in here as an idea (based on the above discussion whether logger(1) is in POSIX and with which options): Couldn't we just have supervise-daemon do the logging for us by adding --syslog-{stderr,stdour} which redirect stdout/stderr to a pipe(3) in the child and then poll that pipe in supervise-daemon's event loop and syslog(3) everything from there in a configurable format?

That should also workaround the "logger may itself crash"-issue which @navi-desu mentioned in #806 (comment). Of cause, the supervise-daemon may itself still crash but that we cannot mitigate with the current architecture anyhow.

[thesamesam]

In this case, enabling logging by default doesn't help in the situation you're describing.

This is a misunderstanding of the point. "Logging configured" here refers to the init script as well. We can assume users have a syslog if they want the logging. If not, not our problem, and they can override it if they wish too.

Option 3 is precisely the one we're concerned about, and the most common scenario from speaking to very confused users and folks writing init scripts.

Okay so how about this: if rc_{output,error}_logger is not set by the user, then instead of statically setting it to logger ... it instead probes if logging is actually active or not and sets a default if it is otherwise doesn't set anything.

I'm not sure if this is robust if the logger isn't up yet or gets restarted.

[amano-kenji]

I'm worried about wear and tear of SSD and HDD, caused by excessive logging. Some services might print a lot. I want to minimize unnecessary logging. In my experiences, SSDs don't last very long.

[thesamesam]

I don't think that's relevant to this bug, sorry. You're free to still configure your syslog daemon appropriately, override the defaults, etc.

Also, daemons tend to be very quiet on std*? The issue here is output being lost when they fail.

[amano-kenji]

In my experiences, some daemons can be verbose on stdout or stderr. At one point, I didn't configure log rotation, so /var became full. If openrc runs on microSD card, minimizing logging will be important. Logging eats up precious write cycles of microSD cards.

I don't like the idea of logging everything by default.

[thesamesam]

You'll have issues with other software already. Note that logger will use syslog and you can configure that appropriately to discard if you wish. Regardless of this change being merged or not, you still need rotation, and still need to setup discarding logs if you don't want them.

Again, this isn't the place, sorry. I don't want to litigate your position on logging overall. The issue here is init scripts being hostile to debugging and valuable information being lost, which often surprises people.

[navi-desu]

while the implementation in this pr isn't quite what's gonna get merged, whatever we merge will be a default -- meaning you'll be able to override it in rc.conf should you be concerned with logfiles

[amano-kenji]

When a service crashed, I looked in man openrc-run and found output_log and error_log. I redirected stdout and stderr to files in /tmp. It worked.

If you want to improve discoverability of logging facility, logging can be documented outside man openrc-run which is kind of obscure.

auditd is one optional logger which can be a sane "documented" default.

https://wiki.gentoo.org/wiki/Openrc can say logger is set to auditd by default. If you want to see logs, add auditd to openrc default runlevel.

[thesamesam]

When a service crashed, I looked in man openrc-run and found output_log and error_log. I redirected stdout and stderr to files in /tmp. It worked.

As we said above, we don't consider needing to do that to debug a failing service to be reasonable for default behaviour.

[amano-kenji]

At least, don't log onto /var/log/messages where everything is bundled together and difficult to find.

[amano-kenji]

One problem with sending everything to syslog is that syslog records everything onto /var/log/messages.

Even though I had a separate log file for each user service, I preferred deleting user service logs before logging in again because I was disoriented by logs from previous login sessions.

There should be an easier way to browse logs. At least, emptty rotates its own logs whenever I log in. Thus, I can see the logs from the current login session. Emptty basically sends logs from the previous login session to archives.

I want to be able to easily dig up current session logs and previous session logs for each user service.

[thesamesam]

That depends on the syslog implementation you choose and its configuration.

[amano-kenji]

One problem with this is the order of shutdown.

If sway shuts down before openrc user services do, some openrc user services will log useless errors that will make users think there are errors when there aren't.

If OpenRC user services quit before sway does, then there is no error.

How can I make sure all (graphical) OpenRC user services quit before exiting sway?

[navi-desu]

eventually running sway as a user service (similar to how niri uses systemd) would work for this

[amano-kenji]

emptty executes sway directly. I don't know how emptty would track sway if it's run as a user service.

[amano-kenji]

Is there an openrc command that waits until all (graphical runlevel) OpenRC user services are shut down?

[thesamesam]

As I've asked several times now, please ask them in a new bug or "discussion" thing on GitHub, not on this PR.

[amano-kenji]

I just executed openrc --user default on a shell to stop graphical runlevel user services. OpenRC waited until graphical user services exited. So, if I want to log out, I can make a keyboard shortcut execute an OpenRC command before quitting sway.

[N-R-K]

@amano-kenji the proper place to ask question is https://github.com/OpenRC/openrc/discussions

[te36de]

Stumbled independently into this issue
(https://github.com/OpenRC/openrc/issues/934#issuecomment-3379310097)](url)

In the first place, there needs to be BY DEFAULT CONFIGURATION logging for any openrc components itself. Specifically openrc-user.sh, to catch errors in the software by the sysadmin without having to tinker around with configs:

As a sysadmin, i do not want to have to do a gigantic strace through OpenRC to catch an issue like this (but i had to do it):

16445 dup2(3, 1) = -1 EBADF (Bad file descriptor)
16445 write(2, "/usr/libexec/rc/sh/openrc-user.sh: line 32: 3: Bad file descriptor\n", 67) = 67

This btw was because of user's shell being tcsh which closed the FD 3 in before.

I thought there is a good amount of system software which simply logs directly to log files, such as software Foo to something like /var/log/Foo/errors.log or the like. I'd expect OpenRC to do this as well by default. No external logger needed.

In any case, HOW TO LOG should not be used as an excuse NOT TO LOG.

Logging of the user started software is another issue all together IMHO. I still have not found any documentation exactly how those openrc user daemons are meant to be managed by the user (for gentoo), so i am not even sure if the software is just meant to be started from ~/.profile or also other ways. That makes it hard to give advice how to log for it.

IMHO, OpenRC can not by default log output from any user started software to system files. I remember managing unix systems for hundreds of students with X11. First of all, you do not want system logfiles to necessarily be readable by such users (simplifies hacking), and of course it would easily fill up disks when users start stuff that logs too much. And if OpenRC is not really brutal ensuring all user started daemons are killed after session terminates, logging will be even more of an issue.

In other words: The logfile for user daemons needs to go ultimately into some user quota file. Like in the user directory. If there is no good precedent how to best do this, then it's perfectly fine to just give some example how to set it up. E.g.: some ~/.profile stub to enable such logging to some ~/.openrc_log/daemon-name.log approach.

[navi-desu]

I still have not found any documentation exactly how those openrc user daemons are meant to be managed by the user (for gentoo), so i am not even sure if the software is just meant to be started from ~/.profile or also other ways

we provide a pam module, and a system wide init script. all of: via pam session using pam_openrc, manually by starting user.$USER (e.g. at boot), or manually by calling openrc --user $runlevel in .profile or by hand, are supported ways to do it

OpenRC can not by default log output from any user started software to system files

if we do end up using logger, it would send those to syslog, and thus syslogd's responsability to split them, so, responsibility of the admin to configure their syslogd properly

however, there are good reasons to not use syslog, the main one being having a deterministic path for logs allows status() functions to print a snippet of logs

Like in the user directory. If there is no good precedent how to best do this

$XDG_STATE_HOME is the place to do it, by the xdg basedirs spec: https://specifications.freedesktop.org/basedir-spec/latest/

[thesamesam]

Please stop writing COMMENTS WITH LOTS OF CAPS.