home January 01, 2017

Pfstat


Pfstat tutorial with pfstat.conf

Pfstat is a tool to make graphs out of the data from the pf firewall logs. It can graph throughput in or out of an interface, the amount of packets blocked, states seen and many other values. Pfstat graphs are especially useful to spot odd behavior like bandwidth over usage and aggressive scanning or DDOS attacks. With interface graphs you can keep an eye on traffic patterns and better understand your network.

Pfstat is based on RRDtool. Check out out RRDtool tutorial and graph examples if you would like to make your own custom graphs.

Getting Started, looking at the pfstat.conf

The following is the configuration file you can use with the latest version of pfstat. It will generate every variation of graphs from the logs that we could think of. It will also make those graphs in daily, weekly and monthly formats to show you long term statistics of your connection. You might not need all of them, but this will give you the chance to see what you want and what you can do without.

NOTE: This configuraion is for Pfstat versions 2.2 and above. Previous versions used a completely different format. If you have not already done so, please upgrade as the newest versions have significantly more functionality.

#######################################################
###  moneyslow.com  pfstat.conf  BEGIN
#######################################################
#
## Bytes in and out plus states
#
collect 1 = interface "em0" pass bytes in ipv4 diff
collect 2 = interface "em0" pass bytes out ipv4 diff
collect 3 = global states entries

image "/var/www/htdocs/pfstat/pfstat_day.jpg" {
        from 1 days to now
        width 980 height 200
        left
                graph 1 "in" "bytes/s" color 0 192 0 filled,
                graph 2 "out" "bytes/s" color 0 0 255
        right
                graph 3 "states" "entries" color 192 192 0
}

image "/var/www/htdocs/pfstat/pfstat_week.jpg" {
        from 1 weeks to now
        width 980 height 200
        left
                graph 1 "in" "bytes/s" color 0 192 0 filled,
                graph 2 "out" "bytes/s" color 0 0 255
        right
                graph 3 "states" "entries" color 192 192 0
}

image "/var/www/htdocs/pfstat/pfstat_month.jpg" {
        from 1 months to now
        width 980 height 200
        left
                graph 1 "in" "bytes/s" color 0 192 0 filled,
                graph 2 "out" "bytes/s" color 0 0 255
        right
                graph 3 "states" "entries" color 192 192 0
}

#######################################################
#
## Pass/Block packets per second In/Out
#
collect 4 = interface "em0" pass packets in ipv4 diff
collect 5 = interface "em0" pass packets out ipv4 diff
collect 6 = interface "em0" block packets in ipv4 diff
collect 7 = interface "em0" block packets out ipv4 diff

image "/var/www/htdocs/pfstat/pfstat-packets_day.jpg" {
        from 1 days to now
        width 980 height 200
        left
                graph 4 "pass in"   "packets/s" color 0 192 0 filled,
                graph 5 "pass out"  "packets/s" color 0 0 255
        right
                graph 6 "block in"  "packets/s" color 255 0 0,
                graph 7 "block out" "packets/s" color 192 192 0
}

image "/var/www/htdocs/pfstat/pfstat-packets_week.jpg" {
        from 1 weeks to now
        width 980 height 200
        left
                graph 4 "pass in"   "packets/s" color 0 192 0 filled,
                graph 5 "pass out"  "packets/s" color 0 0 255
        right
                graph 6 "block in"  "packets/s" color 255 0 0,
                graph 7 "block out" "packets/s" color 192 192 0
}

image "/var/www/htdocs/pfstat/pfstat-packets_month.jpg" {
        from 1 months to now
        width 980 height 200
        left
                graph 4 "pass in"   "packets/s" color 0 192 0 filled,
                graph 5 "pass out"  "packets/s" color 0 0 255
        right
                graph 6 "block in"  "packets/s" color 255 0 0,
                graph 7 "block out" "packets/s" color 192 192 0
}

#######################################################
#
## Inserts, removals and searches per second
#
collect  8 = global states inserts  diff
collect  9 = global states removals diff
collect 10 = global states searches diff

image "/var/www/htdocs/pfstat/pfstat-states_day.jpg" {
        from 1 days to now
        width 980 height 200
        left
                graph 8 "inserts" "states/s" color 0 192 0 filled,
                graph 9 "removals" "states/s" color 0 0 255
        right
                graph 10 "searches" "states/s" color 255 0 0
}

image "/var/www/htdocs/pfstat/pfstat-states_week.jpg" {
        from 1 weeks to now
        width 980 height 200
        left
                graph 8 "inserts" "states/s" color 0 192 0 filled,
                graph 9 "removals" "states/s" color 0 0 255
        right
                graph 10 "searches" "states/s" color 255 0 0
}

image "/var/www/htdocs/pfstat/pfstat-states_month.jpg" {
        from 1 months to now
        width 980 height 200
        left
                graph 8 "inserts" "states/s" color 0 192 0 filled,
                graph 9 "removals" "states/s" color 0 0 255
        right
                graph 10 "searches" "states/s" color 255 0 0
}

#######################################################
#
## Bits per seconds per Queue (for the moneyslow.com PF Example)
#
collect 11 = queue "ack" pass bytes diff
collect 12 = queue "dns" pass bytes diff
collect 13 = queue "ssh" pass bytes diff
collect 14 = queue "bulk" pass bytes diff

image "/var/www/htdocs/pfstat/pfstat-queues_day.jpg" {
       from 1 days to now
       width 980 height 150
        left
               graph 11 bps "ack" "bits/s" color 0 192 192,
               graph 12 bps "dns" "bits/s" color 192 0 192,
               graph 13 bps "ssh" "bits/s" color 255 0 0,
               graph 14 bps "bulk" "bits/s" color 192 192 0
}

image "/var/www/htdocs/pfstat/pfstat-queues_week.jpg" {
        from 1 weeks to now
        width 980 height 150
        left
               graph 11 bps "ack" "bits/s" color 0 192 192,
               graph 12 bps "dns" "bits/s" color 192 0 192,
               graph 13 bps "ssh" "bits/s" color 255 0 0,
               graph 14 bps "bulk" "bits/s" color 192 192 0
}

image "/var/www/htdocs/pfstat/pfstat-queues_month.jpg" {
        from 1 months to now
        width 980 height 150
        left
               graph 11 bps "ack" "bits/s" color 0 192 192,
               graph 12 bps "dns" "bits/s" color 192 0 192,
               graph 13 bps "ssh" "bits/s" color 255 0 0,
               graph 14 bps "bulk" "bits/s" color 192 192 0
}

#######################################################
#
## All other Counters
#
collect 15 = global counters match          diff
collect 16 = global counters bad-offset     diff
collect 17 = global counters fragment       diff
collect 18 = global counters short          diff
collect 19 = global counters normalize      diff
collect 20 = global counters memory         diff
collect 21 = global counters bad-timestamp  diff
collect 22 = global counters congestion     diff
collect 23 = global counters ip-option      diff
collect 24 = global counters proto-cksum    diff
collect 25 = global counters state-mismatch diff
collect 26 = global counters state-insert   diff
collect 27 = global counters state-limit    diff
collect 28 = global counters src-limit      diff
collect 29 = global counters synproxy       diff

image "/var/www/htdocs/pfstat/pfstat-errors_day.jpg" {
       from 1 days to now
       width 980 height 200
        left
               graph 17 "frag" "/s" color 192 0 192,
               graph 22 "cong" "/s" color 0 192 192,
               graph 23 "iopt" "/s" color 0 0 255,
               graph 24 "csum" "/s" color 192 192 0,
               graph 25 "mism" "/s" color 255 0 0
#              others are usually all zero here
        right
               graph 15 "match" "/s" color 0 192 0
}

image "/var/www/htdocs/pfstat/pfstat-errors_week.jpg" {
        from 1 weeks to now
        width 980 height 200
        left
                graph 17 "frag" "/s" color 192 0 192,
                graph 22 "cong" "/s" color 0 192 192,
                graph 23 "iopt" "/s" color 0 0 255,
                graph 24 "csum" "/s" color 192 192 0,
                graph 25 "mism" "/s" color 255 0 0
#               others are usually all zero here
        right
                graph 15 "match" "/s" color 0 192 0
}

image "/var/www/htdocs/pfstat/pfstat-errors_month.jpg" {
        from 1 months to now
        width 980 height 200
        left
                graph 17 "frag" "/s" color 192 0 192,
                graph 22 "cong" "/s" color 0 192 192,
                graph 23 "iopt" "/s" color 0 0 255,
                graph 24 "csum" "/s" color 192 192 0,
                graph 25 "mism" "/s" color 255 0 0
#               others are usually all zero here
        right
                graph 15 "match" "/s" color 0 192 0
}
#######################################################
###  moneyslow.com  pfstat.conf  END
#######################################################

Starting the install

Step 1: Install pfstat from package or from source. For the example we are using the package from OpenBSD which is v2.2.

Step 2: Place the pfstat.conf file from above into your /etc/pfstat.conf. You should backup the default pfstat.conf file for future reference if you want to. Make sure to change the interface from "em0" to your card type if you have a different network interface name.

Take some time and look through the config file and familiarize yourself with the options. For example the section "Bits per second per Queue (for the moneyslow.com PF Example)" might need to be changed to reflect the names of the queues in your pf.conf.

Step 3: Pfstat will make the graphs in jpg form in the web directory of your choice. For our example we placed all of the files in "/var/www/htdocs/pfstat/" so our web server can access them. Make sure your directory structure exists before executing pfstat for the first time.

Step 4: Note that in order to collect interface statistics, interface logging has to be enabled using 'pfctl -l iface' (3.1-stable and prior) or 'set loginterface iface' if your pf.conf (-current). Make sure you have interface logging enabled otherwise the pfstat database will be empty and your graphs will be void.

Executing pfstat

Step 5: Pfstat will run a job every minute to collect information about the interface. This data will be put into the file /var/db/pfstat.db and accessed by the graph generator command and the clean up script.

The best way to run pfstat is from a cron job. There are three(3) commands that need to be run.

This is an example of the crontab you can use.

#minute (0-59)
#|   hour (0-23)
#|   |    day of the month (1-31)
#|   |    |   month of the year (1-12)
#|   |    |   |   day of the week (0-6 with 0=Sun)
#|   |    |   |   |   commands
#|   |    |   |   |   |
#### Generate pfstat graphs
*    *    *   *   *   /usr/local/bin/pfstat -q -d /var/db/pfstat.db
10   6-19 *   *   *   /usr/local/bin/pfstat -p -d /var/db/pfstat.db
25   3    *   *   *   /usr/local/bin/pfstat -t 30 -d /var/db/pfstat.db

In Conclusion

Now that pfstat is configured and running you will now be able to see your graphs. Given some time you will be able to see patterns on your network interface and how it is used through the month. You may notice that network bandwidth drops during the weekends when employees go home, or you might notice a spike in use at night that might call for further investigation.

You have the choice of looking at the jpg's individually, but it might be a better idea to setup a very simple web server and serve out the graphs this way. Check out the moneyslow.com Home Page for instructions on setting up a dead simple web server called tHttpd. It really will take less than five minutes to setup. We also have detailed instructions for Lighttpd and Apache. Once you have a web server up just make a simple html page that points to all of the jpgs.

Reducing the size of the jpg graphs

One of the big problems you may have noticed is the size of the jpg files pfstat uses for graphs. A standard graph which is the width of 980px and a height 200px can be as much as 70 kilobytes. If you have 10 graphs on the page it takes quite a bit of bandwidth and loads slowly especially if you are bandwidth limited.

You can reduce the size of the jpgs significantly by changing the quality of the pictures. If you kept the same amount of graphs and reduced the size of the jpgs from 70KB to 20KB the pages would be more responsive.

Option #1: Using a shell script to reduce the jpg quality

If you installed pfstat in package form then the following script is the easiest way to modify the jpgs. The following script will reduce the jpeg quality of the existing pfstat graphs to 30% of normal and optimize the format. The name of the script is called reduce_pfstat_size.sh . You might want to test out the quality level to suite your needs. 30% is more than clear, but you might prefer another setting. When you are ready to use the script, run it directly after the line "pfstat -p -d /var/db/pfstat.db" which makes the graphs.

#!/bin/csh
#
## moneyslow.com  reduce_pfstat_size.sh
#
## Change to pfstat graph directory
cd /var/www/htdocs/vault/
#
## Reduce the quaility of pfstat jpgs to 30%
foreach file ( pfstat*.jpg )
 djpeg $file | cjpeg -optimize -quality 30 > $file:r.temp
 mv $file:r.temp $file
end

Option #2: Editing the pfstat source to reduce the jpg quality

If you build pfstat from source then you can have it generate the graphs in the quaility you specify. Look in the graph.c file for the following:

if (matrix->type == 0)
     gdImageJpeg(im, out, 95);
        else
     gdImagePng(im, out);

The "95" above is the percentage of quality for the jpeg. Just replace 95 with 30 for a 30% quality level. This hint is courtesy of the technical reference site Pantz.org

Questions?

How log does pfstat take to generate graphs? On a P4 3GHz (1gig ram) running pfstat with 30 days of logs it will generate the graphs in less than 60 seconds. The amount of time pfstat runs depends on the amount of graphs it needs to make and the amount of data in the logs. On a slower system you might want to only generate the necessary graphs and only when you know people are going to access them. For example, during business hours.