Rspamd + Dovecot + ISPConfig: Automatic Spam/Ham Training

How to make Rspamd train spam/ham automatically.

If an email is moved to the IMAP Junk folder, Rspamd should automatically learn it as spam. If the email is moved out of the Junk folder again, it should automatically learned again as ham by Rspamd. The more users do this, the faster the spam filter is trained because the settings apply server-wide.

A correctly installed ISPConfig-server with Rspamd on Ubuntu according to the Perfect Server Setup is required for this tutorial.

Tested with Ubuntu 20 and Debian 12.

Rspamd manual learning

Spam

Example of using a user’s Junk folder for Rspamd spam training (unwanted mails)

rspamc learn_spam /var/vmail/domain.xx/username/Maildir/.Junk/cur
Results for file: cur/1638796375.M239907P2872704.ispconfig.serv.er,S=4305,W=4397:2,Sb (0.035 seconds)
error = "<[email protected]> has been already learned as spam, ignore it";
filename = "cur/1638796375.M239907P2872704.ispconfig.serv.er,S=4305,W=4397:2,Sb";
scan_time = 0.036000;

Results for file: cur/1638786629.M743027P2830658.ispconfig.serv.er,S=57675,W=58742:2,ST (0.055 seconds)
success = true;
filename = "cur/1638786629.M743027P2830658.ispconfig.serv.er,S=57675,W=58742:2,ST";
scan_time = 0.056000;

Ham

Example of using a user’s INBOX for Rspamd ham training (harmless mails)

rspamc learn_ham /var/vmail/domain.xx/username/Maildir/cur

Automatic Rspamd training with Dovecot

Configure Dovecot for ISPConfig

Don’t edit /etc/dovecot.conf directly because it will be overwritten after each ISPConfig update. Edit /usr/local/ispconfig/server/conf-custom/install/dovecot_custom.conf.master instead and do a Force Update. This will reconfigure /etc/dovecot/dovecot.conf and restart Dovecot.

vi /usr/local/ispconfig/server/conf-custom/install/dovecot_custom.conf.master
protocol imap {
  mail_plugins = imap_sieve
}

plugin {
  sieve_plugins = sieve_imapsieve sieve_extprograms

  # From elsewhere to Spam folder
  imapsieve_mailbox1_name = Junk
  imapsieve_mailbox1_causes = COPY
  imapsieve_mailbox1_before = file:/etc/dovecot/rspamd/rspamd-learn-spam.sieve

  # From Spam folder to elsewhere
  imapsieve_mailbox2_name = *
  imapsieve_mailbox2_from = Junk
  imapsieve_mailbox2_causes = COPY
  imapsieve_mailbox2_before = file:/etc/dovecot/rspamd/rspamd-learn-ham.sieve

  sieve_pipe_bin_dir = /etc/dovecot/rspamd

  sieve_global_extensions = +vnd.dovecot.pipe +vnd.dovecot.environment
}

Write dovecot.conf and restart Dovecot

ispconfig_update.sh --force
.
.
.
The following local config override templates were found, be sure to incorporate upstream changes if needed:

/usr/local/ispconfig/server/conf-custom/install/dovecot_custom.conf.master
.
.
.
Restarting services ...
Update finished.

Check Dovecot configuration

dovecot -n
.
.
.
plugin {
  imapsieve_mailbox1_before = file:/etc/dovecot/rspamd/rspamd-learn-spam.sieve
  imapsieve_mailbox1_causes = COPY
  imapsieve_mailbox1_name = Junk
  imapsieve_mailbox2_before = file:/etc/dovecot/rspamd/rspamd-learn-ham.sieve
  imapsieve_mailbox2_causes = COPY
  imapsieve_mailbox2_from = Junk
  imapsieve_mailbox2_name = *
  quota = dict:user::file:/var/vmail/%d/%n/.quotausage
  quota_status_nouser = DUNNO
  quota_status_overquota = 552 5.2.2 Mailbox is full
  quota_status_success = DUNNO
  sieve = /var/vmail/%d/%n/.sieve
  sieve_after = /var/vmail/%d/%n/.ispconfig.sieve
  sieve_before = /var/vmail/%d/%n/.ispconfig-before.sieve
  sieve_global_extensions = +vnd.dovecot.pipe +vnd.dovecot.environment
  sieve_max_actions = 100
  sieve_max_redirects = 25
  sieve_max_script_size = 2M
  sieve_pipe_bin_dir = /etc/dovecot/rspamd
  sieve_plugins = sieve_imapsieve sieve_extprograms
}
.
.
.
protocol imap {
  auth_verbose = yes
  mail_plugins = imap_sieve quota imap_quota listescape
  namespace {
    inbox = yes
    location =
    prefix =
    separator = /
    type = private
  }
}
.
.
.

Ports used for Dovecot and Rspamd

egrep "993/tcp|4190/tcp|11332/tcp|11333/tcp|11334/tcp" /etc/services
imaps             993/tcp                         # IMAP over SSL
sieve             4190/tcp                        # ManageSieve Protocol
rspamd-proxy      11332/tcp
rspamd-worker     11333/tcp
rspamd-controller 11334/tcp

Check running ports

netstat -unplet | egrep "dovecot|rspamd"
tcp     0    0 0.0.0.0:110        0.0.0.0:*    LISTEN    0      19727752   2946685/dovecot
tcp     0    0 0.0.0.0:143        0.0.0.0:*    LISTEN    0      19727789   2946685/dovecot
tcp     0    0 0.0.0.0:4190       0.0.0.0:*    LISTEN    0      19727736   2946685/dovecot
tcp     0    0 0.0.0.0:993        0.0.0.0:*    LISTEN    0      19727791   2946685/dovecot
tcp     0    0 0.0.0.0:995        0.0.0.0:*    LISTEN    0      19727754   2946685/dovecot
tcp     0    0 127.0.0.1:11332    0.0.0.0:*    LISTEN    121    19729074   2946677/rspamd: mai
tcp     0    0 127.0.0.1:11333    0.0.0.0:*    LISTEN    121    19729084   2946677/rspamd: mai
tcp     0    0 127.0.0.1:11334    0.0.0.0:*    LISTEN    121    19729079   2946677/rspamd: mai
tcp6    0    0 :::110             :::*         LISTEN    0      19727753   2946685/dovecot
tcp6    0    0 :::143             :::*         LISTEN    0      19727790   2946685/dovecot
tcp6    0    0 :::4190            :::*         LISTEN    0      19727737   2946685/dovecot
tcp6    0    0 :::993             :::*         LISTEN    0      19727792   2946685/dovecot
tcp6    0    0 :::995             :::*         LISTEN    0      19727755   2946685/dovecot

Create script folder

mkdir -p /etc/dovecot/rspamd

Create sieve scripts

Spam

vi /etc/dovecot/rspamd/rspamd-learn-spam.sieve
require ["vnd.dovecot.pipe", "copy", "imapsieve", "environment", "variables"];

if environment :matches "imap.user" "*" {
  set "username" "${1}";
}

pipe :copy "rspamd-learn-spam.sh" [ "${username}" ];

Ham

vi /etc/dovecot/rspamd/rspamd-learn-ham.sieve
require ["vnd.dovecot.pipe", "copy", "imapsieve", "environment", "variables"];

if environment :matches "imap.mailbox" "*" {
  set "mailbox" "${1}";
}

if string "${mailbox}" "Trash" {
  stop;
}

if environment :matches "imap.user" "*" {
  set "username" "${1}";
}

pipe :copy "rspamd-learn-ham.sh" [ "${username}" ];

Compile sieve scripts

sievec /etc/dovecot/rspamd/rspamd-learn-ham.sieve
sievec /etc/dovecot/rspamd/rspamd-learn-spam.sieve

Compiled sieve scripts were created now in /etc/dovecot/rspamd

find /etc/dovecot/rspamd|grep svbin$
/etc/dovecot/rspamd/rspamd-learn-ham.svbin
/etc/dovecot/rspamd/rspamd-learn-spam.svbin

Set permissions of sieve scripts

chmod u=rw,go= /etc/dovecot/rspamd/rspamd-learn-{spam,ham}.{sieve,svbin}
chown vmail:vmail /etc/dovecot/rspamd/rspamd-learn-{spam,ham}.{sieve,svbin}

Create shell scripts

Spam

vi /etc/dovecot/rspamd/rspamd-learn-spam.sh
#!/bin/sh
exec /usr/bin/rspamc -h localhost:11334 learn_spam

Ham

vi /etc/dovecot/rspamd/rspamd-learn-ham.sh
#!/bin/sh
exec /usr/bin/rspamc -h localhost:11334 learn_ham

Set permissions of shell scripts

chmod u=rwx,go= /etc/dovecot/rspamd/rspamd-learn-{spam,ham}.sh
chown vmail:vmail /etc/dovecot/rspamd/rspamd-learn-{spam,ham}.sh

Check permissions of scripts

ls -al /etc/dovecot/rspamd
total 32
drwxr-xr-x 2 root  root  4096 Dec  6 20:52 .
drwxr-xr-x 5 root  root  4096 Dec  6 19:17 ..
-rwx------ 1 vmail vmail   60 Dec  6 20:48 rspamd-learn-ham.sh
-rw------- 1 vmail vmail  314 Dec  6 19:59 rspamd-learn-ham.sieve
-rw------- 1 vmail vmail  452 Dec  6 20:07 rspamd-learn-ham.svbin
-rwx------ 1 vmail vmail   61 Dec  6 20:49 rspamd-learn-spam.sh
-rw------- 1 vmail vmail  199 Dec  6 19:58 rspamd-learn-spam.sieve
-rw------- 1 vmail vmail  354 Dec  6 20:05 rspamd-learn-spam.svbin

Restart services

service dovecot restart
service rspamd restart

Finished!

From now on, any email that is moved to the Junk folder will be automatically learned as spam by Rspamd. If the message is moved out of the Junk folder again, Rspamd will automatically learn it as ham again.

Rspamd logfile

While moving an email from or to the Junk folder

tail -f /var/log/rspamd/rspamd.log|grep controller
2021-12-06 21:06:39 #2970301(controller) <ccf5c6>; csession; rspamd_controller_check_password: allow unauthorized connection from a trusted IP 127.0.0.1
2021-12-06 21:06:39 #2970301(controller) <ccf5c6>; csession; rspamd_message_parse: loaded message; id: <[email protected]>; queue-id: <undef>; size: 4397; checksum: <e33773d1266d71b688d27d4b2da4d60e>
2021-12-06 21:06:39 #2970301(controller) <ccf5c6>; csession; rspamd_mime_part_detect_language: detected part language: en
2021-12-06 21:06:39 #2970301(controller) <ccf5c6>; csession; rspamd_controller_learn_fin_task: <127.0.0.1> learned message as ham: [email protected]

2021-12-06 21:08:12 #2970301(controller) <a983e5>; csession; rspamd_controller_check_password: allow unauthorized connection from a trusted IP 127.0.0.1
2021-12-06 21:08:12 #2970301(controller) <a983e5>; csession; rspamd_message_parse: loaded message; id: <[email protected]>; queue-id: <undef>; size: 4397; checksum: <e33773d1266d71b688d27d4b2da4d60e>
2021-12-06 21:08:12 #2970301(controller) <a983e5>; csession; rspamd_mime_part_detect_language: detected part language: en
2021-12-06 21:08:12 #2970301(controller) <a983e5>; csession; rspamd_controller_learn_fin_task: <127.0.0.1> learned message as spam: [email protected]

Rspamd GUI

The number of emails learned is constantly increasing.

Rspamd configuration

ISPConfig GUI

Email → Spamfilter-Policy → Normal → Rspamd