Too Many sa-learn Processes Make MiaB Server Unresponsive and Can Crash Mail Processes

[MSSEsq]

When a mail account hosted by a MiaB server has received many junk mail messages and a mail client application processes a large number of those messages in a relatively short period of time, the MiaB server launches many sa-learn processes as the mail user. This often causes all available memory, both real and virtual, to be used, which leaves the server unresponsive with almost no CPU capacity available for user processes. This can even cause processes that process mail to crash. The only way to recover and restore functionality is to restart the system. If there are still many junk mail messages that SpamAssassin must process after a system restart, then many sa-learn processes are launched again and the problem recurs.

I researched SpamAssassin configuration options and did not find a way to limit the number of sa-learn processes that are launched. If there is a way to make MiaB check available free memory and CPU utilization before additional sa-learn processes are launched, and then only allow new sa-learn processes too be launched if adequate system resources are available, then this MiaB system instability would be remedied.

The system being reported about runs on a virtual machine with 2 GB of real RAM and 1 GB of virtual swap memory. Greatly increasing the system resources allocated to the VM avoids this problem, but then that greatly increases the cost of the VM hosting service while the additional allocated system resources are only utilized in the situation described in this problem report. Documentation about sa-learn is available here: sa-learn Manual.

[downtownallday]

Maybe try setting local_destination_recipient_limit (in /etc/postfix/main.cf) greater than 1. Caveat: I don't know if changing it would trigger other issues, though.

See: https://cwiki.apache.org/confluence/display/SPAMASSASSIN/OutOfMemoryProblems

and Postfix settings:

local_destination_concurrency_limit
local_destination_recipient_limit

both of which are at the defaults (2 and 1)

[MSSEsq]

See: https://cwiki.apache.org/confluence/display/SPAMASSASSIN/OutOfMemoryProblems

Thanks for the link and suggestion. Increasing the referenced Postfix limit settings would increase memory requirements, so that would not be helpful. The default settings are very low, so there is only room to lower the concurrency limit from two to one. The out-of-memory problem occurs when a mail client filters mail and moves a large number of messages from the Inbox to the junk mail folder. That causes too many sa-learn processes to be launched. Since the messages were already in the Inbox before being moved, the Postfix local destination limits seem unlikely to have any effect upon the problem.

The Wiki article firmly recommends using spamd which runs SpamAssassin as a daemon and keeps its modules persistently loaded in memory. I searched for Ubuntu spamd packages and found that the minimum supported Ubuntu version is Noble (24.04LTS), so running SpamAssassin as a daemon is not currently possible. Unless there is another current option that I have not found, this problem will persist until MiaB is updated to run on Ubuntu Noble or later.

[downtownallday]

Miab uses spampd. It's a proxy and is always running. It is a replacement for spamd. Mail from postfix is passed to spampd, which forwards to dovecot after performing SpamAssassin functions.

spampd has some command line arguments that might help, eg. --max-requests. See the docs.

[MSSEsq]

spampd has some command line arguments that might help, eg. --max-requests. See the docs.

Thanks for directing me to spampd. Unfortunately, there are no spampd configuration options that could impact the problem that I reported. Interestingly, the current version of spampd, which MiaB does not yet use, has significantly better memory management, but only for spampd processes, which are not at issue.

Since excessive sa-learn processes are at the root of the problem that I reported, I reviewed the MiaB code to determine how sa-learn processes are launched. I found that the MiaB setup script /root/mailinabox/setup/spamassassin.sh configures the Dovecot anti-spam plugin by writing /etc/dovecot/conf.d/99-local-spampd.conf to call /usr/local/bin/sa-learn-pipe.sh with appropriate --spam or --ham options which calls /usr/bin/sa-learn when messages are moved into or out of a Junk mail folder. When a mail client application moves a large number of messages into the Junk mail folder of a mail user account based upon the internal Junk mail filtering of the mail client application, then the anti-spam plugin runs sa-learn-pipe.sh which tries to launch as many sa-learn processes as the number of messages moved. The Ubuntu memory management is unable to handle this when the number of messages moved is large. All memory is quickly used, including all virtual memory, which causes the MiaB System Status Checks to report:

✖ Dovecot LMTP LDA is not running (port 10026).
✖ IMAPS (dovecot) is not running (port 993).
✖ Mail Filters (Sieve/dovecot) is not running (port 4190).

Restarting the MiaB server clears these reported errors, but they quickly recur if the mail client application still needs to move a large number of messages into the Junk mail folder of the mail account.

If I understand this properly, and if the MiaB server is allocated 2 GB of memory (which is often less than half used) then a remedy of this problem seems to require making sa-learn-pipe.sh only launch one sa-learn process for each Junk mail folder that one or more messages are moved to or from. Lines 670–671 of /usr/bin/sa-learn states: “Simply run this command once for each of your mail folders, and it will ''learn'' from the mail therein.” As noted above, it appears that sa-learn-pipe.sh is trying to launch one instance of sa-learn for each message moved rather than one instance for each Junk mail folder to or from which a message has been moved. Therefore, it appears that multiple sa-learn processes are simultaneously processing messages in a single Junk mail folder, especially considering that many more sa-learn processes can be running simultaneously than there are Junk mail folders on the MiaB system. It seems logically possible that a MiaB server could be hosting so many mail accounts that even launching one instance of sa-learn per applicable folder could consume all available memory, so only launching as many sa-learn processes as available system resources can support would be desirable, if practicable.

Such modifications to the MiaB code are beyond my skill level. Since this reported problem causes important Dovecot processes to crash, I hope that this report encourages someone with the requisite skill level to try developing MiaB code modifications that could provide a remedy.

[MSSEsq]

The problem described in this issue report has persisted for years, and has troubled multiple MiaB users. The following links to MiaB Forum discussions provide additional information:

Sa-learn going nuts May, 2021
Found a workaround for sa-learn high CPU usage July, 2021
    References: High CPU usage for "sa-learn" and "spamtrain" processes in a server with Plesk
Sa-learn processes killing the box December, 2022
File explosion in spam folder on server July, 2024

Forum discussion included possibly making the sa-learn process more nice to give it a lower priority for allocation of system resources. Doing that seems likely to mitigate the problem, yet unlikely to remedy it. The discussions describe extreme redundancy of messages, which correlates with the very excessive number of sa-learn processes launched. Since each sa-learn process can process all the mail messages in a mail folder, there should never be more sa-learn processes than folders to be processed. As noted above, many more sa-learn processes can be run at one time than there are mail folders to be processed.

Additional Forum discussions that mention sa-learn are found by this search: Results for sa-learn

[MSSEsq]

I have investigated further by allowing mail to be downloaded from an account that I knew would trigger the problem, then ran top to see the sa-learn processes accumulate toward eventually overwhelming available memory. I then successively ran ps -C sa-learn -o args to see more detail about what was happening. To my surprise, I found that messages were being processed as ham rather than as spam. The following is an example of the command along with what is returned:

root@mail:~# ps -C sa-learn -o args
COMMAND
/usr/bin/perl -T -w /usr/bin/sa-learn --ham /tmp/sendmail-msg-1051524.txt
/usr/bin/perl -T -w /usr/bin/sa-learn --ham /tmp/sendmail-msg-1051528.txt
/usr/bin/perl -T -w /usr/bin/sa-learn --ham /tmp/sendmail-msg-1051535.txt
/usr/bin/perl -T -w /usr/bin/sa-learn --ham /tmp/sendmail-msg-1051541.txt
/usr/bin/perl -T -w /usr/bin/sa-learn --ham /tmp/sendmail-msg-1051549.txt
/usr/bin/perl -T -w /usr/bin/sa-learn --ham /tmp/sendmail-msg-1051553.txt
/usr/bin/perl -T -w /usr/bin/sa-learn --ham /tmp/sendmail-msg-1051559.txt
/usr/bin/perl -T -w /usr/bin/sa-learn --ham /tmp/sendmail-msg-1051568.txt
/usr/bin/perl -T -w /usr/bin/sa-learn --ham /tmp/sendmail-msg-1051572.txt

Further investigation is needed to determine why many simultaneously running sa-learn processes accumulate referencing individual mail messages until system failure occurs due to exhaustion of all available RAM. Lines 670–671 of /usr/bin/sa-learn states: “Simply run this command once for each of your mail folders, and it will ''learn'' from the mail therein.” This seems to indicate that the commands with arguments listed above should end with /tmp rather than with the names of individual message files, and that only one sa-learn process should be running to learn from all the messages in the /tmp directory. Also, it needs to be verified that those messages should be learned as ham rather than as spam.

[kiekerjan]

root@mail:~# ps -C sa-learn -o args
COMMAND
/usr/bin/perl -T -w /usr/bin/sa-learn --ham /tmp/sendmail-msg-1051524.txt

This command indeed looks a sa-learn-pipe was triggered. This is configured to run by dovecot when a message is moved from or to the spam folder.
To me this suggests that it does not matter whether spamd or spampd is used. I think the sa-learn program does not use those programs, but directly works on the spamassassin local database.

I have investigated further by allowing mail to be downloaded from an account that I knew would trigger the problem, then ran top to see the sa-learn processes accumulate toward eventually overwhelming available memory.

Can you elaborate a bit on what exact actions you take, and what are the properties of the account you mention?

I tried the following:

• Accessing the mailbox via Roundcube or Snappy webmail, select multiple messages and move them to the spam folder
  Result: a sa-learn-pipe process is started for each moved message. The processes are started in serial, meaning that only one is running at a time.
• Accessing the mailbox via Roundcube or Snappy webmail, select a single message and move them to the spam folder. Do this for multiple messages.
  Result: a sa-learn-pipe process is started for each moved message. The processes are started in parallel, meaning that they are running at the same time.
• Accessing the mailbox via K9 Android mail app, select multiple messages and move them to the spam folder
  Result: Processes are started in serial
• Accessing the mailbox via K9 Android mail app, select single messages and move them to the spam folder
  Result: Undetermined. I´m unable to move the messages fast enough to determine whether messages are processed in serial or in parallel...

My conclusion is that the trigger towards the sa-learn process should be limited. I wonder if the sa-learn-pipe.sh script can be made smarter, or if there is a setting in Dovecot or in the antispam plugin. While investigating this, I found that the dovecot antispam plugin is quite old (2017), and it might be possible to replace it with IMAPSieve Perhaps this has smarter handling of multiple messages.

Final note: my server now runs Ubuntu 24.04, and thus also Spamassassin 4.0.0 and Dovecot 2.3.21. The official Mail-in-a-Box server has Spamassassin 3.4.6 and Dovecot 2.3.16. There might obviously be differences between those versions.

[MSSEsq]

Can you elaborate a bit on what exact actions you take, and what are the properties of the account you mention?

I tried the following:

• … The processes are started in serial, meaning that only one is running at a time.
• … Result: a sa-learn-pipe process is started for each moved message. The processes are started in parallel, meaning that they are running at the same time.…

My conclusion is that the trigger towards the sa-learn process should be limited. I wonder if the sa-learn-pipe.sh script can be made smarter, or if there is a setting in Dovecot or in the antispam plugin.

The only action I took was launching Mail on macOS Sequoia 15.6.1 and allowing it to download a large number of mail messages from our MiaB server. The Mail account has junk mail filtering enabled which is set to move junk mail messages to the Junk mailbox, to trust junk mail headers in messages, and to filter junk mail before applying rules that direct matching messages to specific mailboxes.

Your testing results indicate that the mail system is stressed by sa-learn processes running in parallel when a mail client application moves many mail messages in parallel rather than serially.

The MiaB SpamAssassin setup script, spamassissin.sh, references http://wiki2.dovecot.org/Plugins/Antispam, which is no longer published. Current documentation is at: Dovecot Community Edition (CE) Spam Reporting. The page at that link provides useful information, including:

WARNING
You cannot run shell scripts anywhere you want.

Sieve allows you to only run scripts under sieve_pipe_bin_dir. You can't use /usr/local/bin/my-sieve-filter.sh, you have to put the script under sieve_pipe_bin_dir and use my-sieve-filter.sh in the script instead.

The MiaB SpamAssassin setup script, spamassissin.sh, configures the antispam plugin to execute /usr/local/bin/sa-learn-pipe.sh which runs usr/bin/sa-learn, which is contrary to this warning.

Local Reporting
In this setup, the sieve scripts pass the reported message to local binaries to do classification.…

With this configuration, moving mails will slow down due to learn being done per email. If you want to avoid this, you need to think of something else. Probably piping things into a FIFO or perhaps using a socket based worker might work better.…

Shell Scripts

SpamAssassin

You can also use sa-learn.

Note that using sa-learn often incurs significant start-up time for every message. This can cause "lockout" of the user until all the processes sequentially complete, potentially tens of seconds or minutes.…

I have witnessed the sa-learn processes take days consuming 100% of multiple CPU cores to complete if the MiaB server is temporarily allocated extensive compute resources.

The current documentation includes script examples, including for the use of sa-learn:

sa-learn-spam.sh

#!/bin/sh
exec /usr/bin/sa-learn -u ${1} --spam

and

sa-learn-ham.sh

#!/bin/sh
exec /usr/bin/sa-learn -u ${1} --ham

These examples use sa-learn differently than /usr/local/bin/sa-learn-pipe.sh does. The examples do not include file names as arguments. That corresponds with the sa-learn manual which states: “Simply run this command once for each of your mail folders, and it will ''learn'' from the mail therein.” This seems to indicate that “/sendmail-msg-$$.txt” should be deleted from /usr/local/bin/sa-learn-pipe.sh or that separate spam and ham scripts should be used as shown in the examples.

[lemanschik]

See this as solved i upgraded MiAB and fixed stuff like that on ubuntu 24.04 will init PR as soon as all is rounded up. FYI spamassasine is now spamd in 24.04 i did also need to take the default config from that to get all working on 24.04 keep you updated i have this issue on my list.

[lemanschik]

Hot fix if your box crashes now!

In /etc/dovecot/conf.d/99-local-spampd.conf

protocol imapsieve {
  # Temporarily comment out the following line to disable sa-learn on message moves
  # mail_plugins = $mail_plugins antispam
}

Caveat

this will stop SpamAssassin from learning from their actions, but it will restore server stability. They must remember to re-enable it after an official fix is released.

Longterm and mid term fix are still in progress.

[MSSEsq]

[T]his will stop SpamAssassin from learning from their actions, but it will restore server stability. They must remember to re-enable it after an official fix is released.

Thanks for documenting a simple temporary workaround.

I recently studied mail logs and reviewed mail message copies stored temporarily on our MiaB server awaiting processing by SpamAssassin. I discovered that a single promotional news message was being sent to our MiaB server by a Mailchimp server a huge number of times over two months. Each copy was being processed by SpamAssassin, which greatly exacerbated the problem of too many simultaneous sa-learn processes. Reporting the redundant message transmissions to Mailchimp accomplished nothing. Mail logs documenting repetitive communication sessions initiated by Mailchimp servers from an IP address directly assigned by ARIN to Mailchimp were disregarded.

The news message included within the header a coded email address for return of bounced messages. I programmed a mail filter using the MiaB Webmail GUI that redirects the repetitively sent news message to the Mailchimp bounce address. That now stops those message copies from reaching SpamAssassin. With the extra processing burden of the messages sent by Mailchimp now mitigated, our MiaB server has been running reasonably well with 2 GB of RAM allocated. Although Mailchimp should never send hundreds of thousands of copies of a particular promotional email message to a single email address, the outsized stress put upon our MiaB server brought a weakness to the fore that long warranted attention. I thank you for your continuing work toward a permanent fix for the underlying MiaB problem. Sometimes dark clouds have silver linings.

[myfirstnameispaul]

Anecdotal, but I've had success with using dnswl.org. I think this is because the reputation history is used by a great many open source tools (including Spamassassin) and large organizations.