Spamd tarpit and greylisting daemon

Greytrapping is the seeding of an email address so that spammers can find it, but normal users can not. If the email address is used then the sender must be a spammer and they are black listed.

Lets use the email address "greytrap@your_domain.org" as an example. If we took this email and put it in the source HTML of our web site normal humans would not see it. Spammers on the other hand use web page scrapers and bots to harvest email address; so they would find this address.

When the spammer sends mail with the destination address of "greytrap@your_domain.org" spamd knows this is a spammer and SPAMTRAP's them. When a host that is currently greylisted attempts to send mail to a spamtrap address, it is spamtrapped for 24 hours by adding the host to the spamd spamlist <spamd-greytrap>.

You can add as many spamtrapped email address as you want. We even suggest looking through the logs and seeing what invalid addresses spammers are sending mail to. You can then add these addresses too.

To enter a greytrap email into the spamd database use the following format. You can add multiple spamtrap email addresses.

spamdb -T -a 'greytrap@your_domain.org'

To seed the SPAMTRAP email address in HTML use the following format. It looks like a real email address to the spammer's bots, but it will not be visible to any humans looking at the web page.

<a href="mailto:greytrap@your_domain.org"></a>

Optional: Greytrapping all but allowed domains and email addresses

We highly suggest setting up a pre-verified list of email addresses that are known to be valid. Any email server sending mail to an address not on the list must be a spammer; or a confused human. Either way this is a very powerful feature to use and is highly recommended.

The file /etc/mail/spamd.alloweddomains can contain a list of verified domains and addresses which are allowed to go through the grey listing process. They are _not_ whitelisted, just allowed to prove they are good hosts by going through the grey list process. Think of it as a white list of acceptable domains or emails this machine will be willing to receive. If a remote machine is not sending mail to a valid address listed in our spamd.alloweddomains then they should not be contacting us.

spamd.alloweddomains can be used to specify a list of domain name suffixes or full email addresses which must match each destination email address in the grey list. Any destination address which does not match one of the suffixes listed in spamd.alloweddomains will be trapped, exactly as if it were sent to a spamtrap address. When a host that is currently greylisted attempts to send mail to a spamtrap address, it is blacklisted for 24 hours by adding the host to the spamd blacklist <spamd-greytrap>.

In our pf example above we only have two rules, spamd-white and not (!) spamd-white. Since a host in the spamd-greytrap table is not in spamd-white then it will be stuttered at for as long as the remote machine is connected.

This is one of the most powerful options in spamd. If you can keep a list of valid email addresses current in the spamd.alloweddomains file then all other attempts will be grey trapped. At moneyslow.com, this grey trapping option traps over 99% of the spammers and leaves our real mail server to handle real mail.

For example, if spamd.alloweddomains contained the following:

@your_domain.org
critical.com
mary@your_domain.org

The following addresses will _NOT_ be spam trapped:

mary@your_domain.org
frank@your_domain.org
bob@critical.com
bobby@nobby.critical.com
robert@whoop.critical.com

The following address _WILL_ be spam trapped:

baker@test.your_domain.org     - invalid sub domain
marge@machine.your_domain.org  - invalid sub domain
bob@wrong_name.com              - not your hostname

Starting Spamd in grey list mode

To get spamd working you can reboot the box and make sure that the spamd processes have started. You should see the user "_spamd" running on ports 8025 and 8026. If you run "ps -aux | grep spamd" you should see the following processes listed:

root@machine: ps -aux | grep spamd
_spamd   22041  0.0  0.1  9804  1160 ??  Is    Mon11AM    0:01.59 spamd: (pf <spamd-white> update) (spamd)
_spamd   22052  0.0  1.0  9748 10276 ??  S     Mon11AM    0:01.99 spamd: [priv] (greylist) (spamd)
_spamd   22063  0.0  0.1  9848  1236 ??  S     Mon11AM    0:00.14 spamd: (/var/db/spamd update) (spamd)
_spamd   22074  0.0  0.1   560  1124 ??  Ss    Mon11AM    0:02.29 /usr/libexec/spamlogd -I -i lo0

Grey list mode activated

Now, all you have to do is wait for remote mail servers to connect and get grey listed. Use the command "spamd" to list out the database and watch as the hosts get trapped. When a valid mail server connects after the grey list time (25 minutes) and before the greylist timeout (4 hours) they will be whitelisted. Those host will then connect directly to the real mail server. All other hosts who do not connect again will have their entries delete from the database and they will have to start the process all over again.

IMPORTANT NOTE: web mail sites that use multiple mail servers to deliver one email

Web mail services like Google, Hotmail and some other sites use multiple email servers. This would not normally be a problem for grey listing except those services randomly attempt to deliver a single email from any of the mail servers in the pool. The first attempts comes from server_1 then the second from server_3 and the third might be attempted from server_1 again. This causes a problem with grey listing because it is possible that none of the mail servers will be used more than a few times to try to deliver the email. Thus, none of the servers will get white listed.

To fix this problem we have a few options:

Normal Volume Site: If your site gets enough emails from services like Google, Hotmail, etc then they will eventually get whitelisted due to the amount of mail being sent. You would really only need 50-100 emails an hour to be sent to your mail server for many of the Gmail servers to eventually be whitelisted within an hour or so. With enough email grey listing is not a problem.

Low Volume Site: If your mail server is a personal server and you do not get a lot of emails then you may have a problem with Google, Hotmail, etc. You also have a few options available.

Option 1: You could manually whitelist the servers according to the MX records registered to the services. You would only need to white list less than 50% of them as the odds are that a single email will hit at least one of the whitelisted servers during attempted delivery. The rest of the servers will be whitelisted automatically over time.

Option 2: You could watch or have a script monitor the logs and send yourself an alert if one of the servers in question connect. This is more hands on, but it should only take a week or so to find all of the ip in question.

Option 3: Once they are white listed you can extend the whitelist time of 864 hours (-G25:4:864) with a script. If you grep for all the white listed entries and just add the ips again using "spamdb -a "ip" it will extend the whitexp time another 36 days (864 hours). Here is a shell script called "spamd_extend_whitelist.sh" that will list out the WHITE ips from spamdb and refresh their white expire time to 36 days. Run it in a cron job on the first of the month to, in effect, have a permanent white list. FYI: every time you use "spamdb -a" on an ip it looks like another new email was delivered according to spamd. In effect, the last column in the spamdb database updates by one.

#!/bin/sh
#
## moneyslow.com .:. spamd_extend_whitelist.sh
#
for i in `spamdb | grep WHITE | awk -F "|" '{print $2}'`;
 do
  /usr/sbin/spamdb -a $i
done

Option 4: If the white listed ips are always going to be allowed then setup another table in Pf, put the white listed ips in there and always forward them to the real mail server.

Option 5: You could setup your own account on the free services and send yourself 10 emails at once. Looking at the spamd database you will see all of the ips that connect and they should get white listed normally. If not, just manually white list them.

How about a script? White listing using "spamd_whitelist.sh"

Whitelist script: spamd_whitelist.sh :If you wanted to create a whitelist of the most common grey list friendly mail servers that would also be a good option. Here is a script that will dig the host names of the listed domains and make a text file with the results.

#!/bin/sh
#
## moneyslow.com  spamd_whitelist.sh
#

FILE=spamd-spf.txt

rm -f $FILE
touch $FILE

### This first list are for domains who have SPF records.
for domain in \
        aol.com \
        apple.com \
        amazon.com \
        s._spf.ebay.com \
        m._spf.ebay.com \
        p._spf.ebay.com \
        p2._spf.ebay.com \
        c._spf.ebay.com \
        gmx.net \
        _spf.google.com \
        spf-a.hotmail.com \
        spf-b.hotmail.com \
        spf-c.hotmail.com \
        spf-d.hotmail.com \
        _spf-a.microsoft.com \
        _spf-b.microsoft.com \
        _spf-c.microsoft.com \
        southwest.com
do
echo \#$domain >> $FILE;
dig $domain TXT +short | tr "\ " "\n" | grep ^ip4: | cut -d: -f2 | cut -d \" -f 1 >> $FILE;
done

### This second list is for domains you specifically want to allow
### that may not be publicly available or do not have SPF records.
echo \#privatelist >> $FILE;
for privatelist in \
         10.0.0.0/8 \
         127.0.0.0/8 \
         172.16.0.0/12 \
         192.168.0.0/16
do
echo $privatelist >> $FILE;
done

Output of spamd_whitelist.sh would look similar to this:

#aol.com
152.163.225.0/24
205.188.139.0/24
205.188.144.0/24
205.188.156.0/23
205.188.159.0/24
64.12.136.0/23
64.12.138.0/24
64.12.143.99/32
64.12.143.100/32
216.34.51.0/24
#_spf-a.microsoft.com
216.99.5.67
216.99.5.68
207.46.50.82
#_spf-b.microsoft.com
131.107.65.22
217.77.141.52
217.77.141.59
#_spf-c.microsoft.com
203.32.4.25
131.107.70.12
131.107.70.16
86.61.88.25

...many more lines...

Now that you have a list of hosts you can add them all as white list entries to a PF table we will call "company-white". We have to make a PF table because we can not add CIDR-formatted network blocks (i.e. 192.168/16 or 10/8) to spamdb. Using the following few lines in pf will populate the table, redirect traffic and pass the packets to the mail server.

################ Tables ####################################
 table <company-white> persist file "/tools/pf_company_whitelist"

################ Filtering #################################
 pass in log on $ExtIf inet proto tcp from <company-white> to ($ExtIf) port smtp flags S/SA synproxy state rdr-to lo0

Example 2: Run Spamd with the sole purpose of annoying spammers

If you do not run a real mail server, but wish to annoy spammers then using spamd in blacklist mode is perfect. It will trap any connection sent to port 25 of your ip/host and stutter the connection. This will waste the time and queue memory of the remote system. The theory being, if you do not have a real mail server then why would a host connect to you unless they are a spammer.

Add the following lines. The line "spamd_black=YES" turns black list only mode on.

## Add the following to /etc/rc.conf.local
#
spamd_flags="-4 -G25:4:864 -h your_hostname.org -l127.0.0.1 -n \"Sendmail 8.11.4/8.11.1\" -S10 -s1 -v -w1"
spamd_black=YES
spamlogd_flags="-I -i lo0"

These will start spamd with the following options:

• -4 - return error code "451 Temporary failure, please try again later." to the spammer. This is the default option.
• -G25:4:864 - adjust the three time parameters for grey listing to pass time of 25 minutes, greyexp to 4 hours, and white expire time to 864 hours, approximately 36 days
• -h your_hostname.org - tell yet to be whitelisted hosts your hostname is "your_hostname.org"
• -l127.0.0.1 - listen on local host only
• -n \"Sendmail 8.11.4/8.11.1\" - tell yet to be whitelisted hosts your email server is this string. Since spammers are malicious it is best to annonomize your mail server. You can put anything here like a false mail server name or even a something making fun of spammers.
• -S10 - stutter at greylisted/blacklist hosts for the first 10 seconds of the connection (allowed 1-90)
• -s1 - stutter at a speed of 1 character per second (allowed 1-10)
• -v - verbose logging to the "daemon" log file. This will tell spamd to report the From: and To: fields in the logs.
• -w1 - window size of one(1) byte (i.e. on character like a single letter or number). The window size is the amount of outstanding data which is unacknowledged by the recipient and can remain in the network between the client and server. After the client sends this amount of data to spamd, the sender stops and waits for acknowledgment back from spamd (the receiver) that the daemon has received that data and to continue sending the next batch of data.

Setting up pf

Now, add the following to your pf.conf. All these rules do is direct all connections from any remote host to port 25 on your machine directly to spamd. There they will be stuttered at and their time wasted.

## Add the following to /etc/pf.conf
#
# Spamd ( external mail attempts to spamd server )
 pass in log on $ExtIf inet proto tcp from any to ($ExtIf) port smtp flags S/SA synproxy state rdr-to lo0 port spamd
#

Checking /etc/mail/spamd.conf

The spamd.conf file is used to list out black list files. We do _not_ suggest using blacklists because of the unknowns involved in their collection. We do not know what affiliations the people collecting the ips may have or their agenda. To be safe we suggest avoiding blacklists. Spamd in blacklist mode stutters all connections so this is really a moot point.

The following is all you need in your /etc/mail/spamd.conf as to void out all black lists.

all:\
        ::

Starting Spamd in black list mode

To get spamd working you can reboot the box and make sure that the spamd processes have started. You should see he user "_spamd" running on port 8025. If you run "ps -aux | grep spamd" you should see the following processes listed:

root@machine: ps -aux | grep spamd
_spamd   22063  0.0  0.1  9848  1236 ??  S     Mon11AM    0:00.14 spamd: (/var/db/spamd update) (spamd)
_spamd   22074  0.0  0.1   560  1124 ??  Ss    Mon11AM    0:02.29 /usr/libexec/spamlogd -I -i lo0

Black list mode activated

Now, all you have to do is wait for remote mail servers to connect and they will be blacklisted. If you use the program "pftop" or use the command "systat states" you can watch the connections.

What can I expect of spamdb in the real world ?

We have been using spamdb for years. Spamdb is a the gatekeeper you must go through before you get to the real mail server. On average we receive hundreds of smtp connections per day; thousands of attempts per week. These are some of the patterns we have noticed.

The average spammer works week days just like the people they are trying to send mail to. Monday is the day most spammers will start their assault. On average, the most spam is delivered on Monday and Tuesday. The volume will start to drop off on Wednesday and continue as we head towards Friday. Spam on the weekend is lower volume and may be the left over noise from bots still trying to deliver mail.

Here we see a graph of the spamd traffic for one month. We are graphing the number of connections per day. Notice the pattern of high traffic at the beginning of the week and how it drops down towards the weekend.

                    Spamd traffic for a Month
250|                                               *
   |     *             *                           *
200|   * *   *         *           * *           * * *
   |   * * * *       * * * *       * * *   *     * * * * *
150|   * * * *       * * * * *     * * * * *     * * * * * *
   | * * * * * * *   * * * * * *   * * * * * * * * * * * * *
100| * * * * * * * * * * * * * * * * * * * * * * * * * * * *
   | * * * * * * * * * * * * * * * * * * * * * * * * * * * *
50 | * * * * * * * * * * * * * * * * * * * * * * * * * * * *
   |---------------------------------------------------------
     S M T W T F S S M T W T F S S M T W T F S S M T W T F S
         week 1        week 2        week 3        week 4

The reason spammers have almost the same hours as normal people is they need to send out the latest round of scams and have servers accept them before they are blocked by the admins or banned by RBLS. The spammers have a limited window to send out their junk mail and move onto the next scam. If a spammer can send out mail and have a victim accept it right away that gives less time for anti-spam checks to evolve to block them. This is why spam is normally not sent on the weekend. By the time Monday rolled around the anti-spam groups would have a large enough sampling of the spam to block it before the work week started.

On average, 20% of the ip addresses that are trapped return to be stuttered at again and again by spamd. The other 80% either never return or never complete the entire smtp handshake process. One item to note is that many spam scanners machines (especially from Asia) only connect for a few seconds before disconnecting. Real mail servers would never do this. So, if you setup your own real mail server, like postfix, sendmail or qmail, to delay talking to a smtp host for, lets says 60 seconds after they connect, a real mail servers will stay around and spammers will more then likely disconnect.

Spammers get their email address lists from other spammers, scrapper bots looking at web sites and bulk "verified" email databases they can buy. Most of the time the "verified" email address are just made up emails that point to your domain. There is no honor among thieves. These types of smtp connections are easy to deny using a spamd.alloweddomains white list of known emails you will accept. If the email does not match one of the entries in the list the ip is trapped.

Other times the spammer has your actual email address. You can stutter at them, but eventually they may get whitelisted. You may want to increase your greylist time from the default 25 minutes to 30, 45 or 60 minutes. If the spammer gets past spamd they will deliver their spam to you. You then have a small list of ips of the spammer systems and can manually add those ips to a blacklist. You can also use a real-time black list (RBLS) such as Spamhaus.org" on your real mail server. We have an example Postfix mail server that uses quite a few anti-spam measures at our Postfix config "how to" (main.cf).

Blocking the majority of spam is really not too hard. Blocking 99% of it takes a little bit of work and understanding. With spamd you can easily block the zombie spam bot machines and with a competently setup mail server you can deny the rest.

Questions?

Clients connecting to Spamd are overloading my network connection! What can I do?If spamd is using to much bandwidth by sending too many packets try increasing the stutter time "-S90" and the stutter speed "-s5". For example, 600 connections total is 600 packets per 5 seconds. The network would need to handle 120 packets average per second each way; around 100 kilobytes in upload bandwidth. This might still be too much bandwidth for your connection, but you could increase the values as needed or decrease the amount of connections spamd will accept with maxcon "-c". Maxcon may not exceed kern.maxfiles - 200, and defaults to 800. Reduce maxcon to 200 or 100. Figure out how much bandwidth you can afford to use on spamd. Each packet sent from spamd is no more than 83 bytes; 53 bytes payload and 23 bytes for the tcp header. If you want to spam trap 100 clients at 1 packet per second each then you will need (0.83*100)/1 = 83 kilobytes per second. If you increased the stutter time to 5 seconds then you would need (0.83*100)/5 = 16.6 kilobytes per second. You could also put spamd traffic in a lower priority queue and limit total spamd bandwidth. Take a look at our how to for (HFSC) Hierarchical Fair Service Curve Packet Scheduler for more information.

Do you have a script I can run to tally the results of Spamd?Yes. We have a script called Spamd Statistics that will make a HTML page of the results. It will go through the logs and tally all of the time spammers wasted, the amount of bandwidth your host used to respond to spammers and more.

How do I list out the Spamd database?Use the command "spamdb" or sort the list with "spamdb | sort".

How can I change the log file for spamd (/var/log/daemon to /var/log/spamd)?Spamd sends log messages to syslogd using facility daemon and, with increasing verbosity, level err, warn, info, and debug. The following syslog.conf section can be used to log connection details to a dedicated file:

!spamd
daemon.err;daemon.warn;daemon.info  /var/log/spamd

How about using the Greyscanning script by Bob Beck?The grey scanning perl script will watch the spamd logs and make decisions on hosts that connect. We highly suggest at least taking a look at the script as we find it an incredibly powerful tool. It will "process spamdb output and look for patterns. Mainly we look at the greylist entries, and make some decisions about them. if they look excessively spammish, we take some action against them, by running spamdb -t -a to add them as a TRAPPED entry to spamd, meaning spamd in greylist mode will tarpit them for the next 24 hours." Bob Beck's greyscanner.pl

How can I extract just the ip addresses from spamdb?

spamdb | grep WHITE | awk -F "|" '{print $2}'

Do you have a script to show my spamdb whitelisted ips and statistics ?This script will show the a numerically sorted list of white listed ip address and their related host names. Also, there is a count of WHITE, SPAMTRAP and TRAPPED entries. We use this to quickly identify the top spammed email addresses to our domain and top spammer ips. This script simply cats the /var/log/daemon log file. If you setup your server to rotate logs every week for example, then when you run spamdb.sh it will show you stats for the week.

#!/bin/sh
#
## moneyslow.com .:. spamdb.sh
#

if [ $# -eq 0 ]
   then
    echo ""
    echo "  moneyslow.com    spamdb.sh $arg "
    echo "--------------------------------------"
    echo "show    = all whitelisted ips with hostnames"
    echo "stats   = ips in all tables"
    echo ""
   exit
 fi

if [ $1 = "show" ]
   then
    echo " "
     for i in `spamdb | grep WHITE | awk -F "|" '{print $2}' | sort `;
      do
       echo -n $i; echo -n -e "\t"; host $i | awk '{print $5}'
      done
    echo " "
   exit
 fi

if [ $1 = "stats" ]
   then
     echo "moneyslow.com .:. spamdb.sh stats"
     echo " "
     echo "Spamd incoming smtp connections"
     echo -n "  grey list attempts         = ";cat /var/log/daemon | grep disconnected | grep -vc spamd-greytrap
     echo -n "  spamd-greytrap tar pits    = ";cat /var/log/daemon | grep disconnected | grep -c spamd-greytrap
     echo -n "  total connections to spamd = ";cat /var/log/daemon | grep -c disconnected

     echo " "
     echo "Spamd statistics"
     echo -n "  hours spammers have wasted = ";cat /var/log/daemon | grep disconnected | awk '{s += $9} END {print s/3600}'

     echo " "
     echo "Spamdb database classes"
     echo -n "  SPAMTRAP trigger addresses = ";spamdb | grep -c ^SPAMTRAP
     echo -n "  TRAPPED black listed ips   = ";spamdb | grep -c ^TRAPPED
     echo -n "  WHITE listed and clear ips = ";spamdb | grep -c ^WHITE

     echo " "
     echo "Top 10 To: email addresses"
     cat /var/log/daemon | grep " spamd" | grep ": (" | awk '{print $10}' | sort | uniq -c | sort -r | head -10 | tr -d \<\>

     echo " "
     echo "Top 10 remote mail server ips"
     cat /var/log/daemon | grep disconnected | awk '{print $6}' | sort | uniq -c | sort -r | head -10 | tr -d \:
     echo " "
   exit
 fi

What is the maximum white expire (whiteexp) time I can set?864 hours is the upper limit. You can not set the white expire time (-G25:4:864) above this. Take a look for the script "spamd_extend_whitelist.sh" in the examples above. It will 'touch' all of the white listed spamdb entries and re-expire them in another 36 days. Run this script in a cron job on the first of the month and you will have a permanent white list.

How can I manually WHITE list ips addresses in the database?

spamdb -a 127.0.0.1

How can I manually remove ips addresses in the database?

spamdb -d 127.0.0.1

How can I add a SPAMTRAP email address?

spamdb -T -a 'spamtrap@domain.org' 

How can I manually TRAP (i.e. spamd will ignore) an ip for 24 hours?

spamdb -t -a 127.0.0.1

How can I convert Unix time to standard date format ?

date -r 1200337384

Where can I find more information about Spamd?

man spamd  ...and... man spamdb