home January 01, 2017

Pound


a secure reverse proxy

Pound is a reverse-proxy load balancing server. It accepts requests from HTTP/HTTPS clients and distributes them to one or more Web servers. The HTTPS requests can be decrypted and passed to the back-ends as plain HTTP (ssl and load balancing). Pound itself does not serve content but acts as a front end to servers that do. It is a fast, secure and stable frontend for a web server of web cluster.

If more than one back-end server is defined, Pound chooses one of them randomly, based on defined priorities. By default, Pound keeps track of associations between clients and back-end servers (sessions).

Using Pound as a security guard

Most web servers are made to serve out data. It is their purpose for being. They are not designed from the ground up with security in mind. Lighttpd, Apache, tHttpd, Mongrel and others are no exception. With a reverse proxy like Pound, you can deny malicious requests your web server may not be designed to filter out. By separating out bad clients with a separate process like Pound this allows your web server to focus on serving data to well behaving clients.

HELPFUL HINT: You may want to check out Nginx as a front end proxy. It is faster, more efficient and more configurable. Take a look at our Nginx "how to" - Fast and Secure Web Server for more information.

Building Pound

Pound is normally built from source and requires that you also have a version of OpenSSL built with pthreads enabled.

Step 1: Download and build OpenSSL -Using the following set of commands we will change into /tmp and download the latest version of OpenSSL, untar it and change to the untared directory. We then need to configure OpenSSL to use pthreads. Then "make" the binaries, test the OpenSSL functionality (make test) and then install OpenSSL. Be default OpenSSL will install into /usr/local/ssl/ .

cd /tmp
wget http://www.openssl.org/source/openssl-0.9.8g.tar.gz
tar zxvf openssl-0.9.8g.tar.gz
cd openssl-0.9.8g
./config -pthreads
make
make test
make install

Step 2: Download and build Pound -The following commands will build Pound with the version of OpenSSL (pthreads) we built before. Here we will change to /tmp, download the latest version of Pound, untar it and change to the untared directory. Then use ./configure with the "--with-ssl=/usr/local/ssl/" flag to point to our source build of OpenSSL. Then "make" and "make install". The pound binary will be put into /usr/local/sbin by default.

cd /tmp
wget http://www.apsis.ch/pound/Pound-2.4f.tgz
tar zxvf Pound-2.4f.tgz
cd Pound-2.4f
make clean
./configure --with-maxbuf=1024 --with-ssl=/usr/local/ssl/
make
make install

SECURITY: A web proxy can add an extra layer of security to your web cluster. Even if you only have one web server, a proxy can help weed out the malicious clients and bots and scanners; leaving only valid requests for your web server.

The pound config file

In the following scrollable window is the configuration file used with pound. Below the window is an explanation of ever line used in the config. The config file tells pound what ip/port to listen on and how to filter those requests before forwarding to the back end server(s). This is a fully working config file and you are welcome to copy/paste the following and save it to /etc/pound.conf or in any other location you prefer.

####################################
#### moneyslow.com  Pound.conf   BEGIN
####################################
User        "pound"
Group       "pound"
LogFacility daemon
LogLevel    4
Alive       30
Client      10
TimeOut     10
Grace       10

ListenHTTP
   Address    127.0.0.1
   CheckURL   "(^\/|\.html|\.css|\.jpg|favicon\.ico|robots\.txt|\.png)$"
   HeadRemove "X-Forwarded-For" 
   MaxRequest 1024 
   Port       8081
   xHTTP      0

   Err414 "/var/www/htdocs/error/generic_error_page"
   Err500 "/var/www/htdocs/error/generic_error_page"
   Err501 "/var/www/htdocs/error/generic_error_page"
   Err503 "/var/www/htdocs/error/generic_error_page"

   Service
      HeadRequire "(Host: your_host.com|Host: www.your_host.com)"
      BackEnd
         Address  127.0.0.1
         Port     8080
      End
      Emergency
       Address    127.0.0.1
       Port       8888
      End
   End
End
####################################
#### moneyslow.com  Pound.conf     END
####################################

What the pound.conf directives mean

User "pound" and Group "pound" are the username and groupname pound will run as. It is wise not to run any publicly available service as root. Here, we made a user named "pound" that the daemon pound will drop privileges to after loading the directives in the config file. When you start the daemon as "root" it will read the config and then drop privileges and run as the user "pound". (Default: the use who executes the daemon)

LogFacility daemon is the log facility in syslog the pound access and error logs will goto. (Default: daemon)

LogLevel 4 is for extended combined Apache type logging. Specify the logging level: 0 for no logging, 1 (default) for regular logging, 2 for extended logging (show chosen backend server as well), 3 for Apache-like format (Combined Log Format with Virtual Host), 4 (same as 3 but without the virtual host information) and 5 (same as 4 but with information about the Service and BackEnd used). This value can be overridden for specific listeners. (Default: 1)

Alive 30 specifies how often Pound will check for resurrected back-end hosts (Default: 30 seconds). If a back end server goes down, pound will check for the host every 30 seconds to see if it is back up. Make sure to set this low, but not too low. If set too low pound will start using excessive resources checking for downed back end hosts.

Client 10 specifies for how long Pound will wait for a remote client request (default: 10 seconds). After this long has passed without the client sending any data Pound will close the connection. Set it higher if your clients time-out on a slow network or over-loaded server, lower if you start getting DOS attacks or run into problems with IE clients. This value can be overridden for specific listeners. (Default: 10 seconds)

TimeOut 10 is how long should Pound wait for a response from the back-end (in seconds). Default: 15 seconds. This value can be overridden for specific back-ends. (Default: 15 seconds). The timeout value is the maximum time Pound will wait for input (start, middle or end of the request or response). In other words, if the first part of the response arrives and then there is a delay longer than timeout seconds before the next part (packet) is received, the transaction fails. Same applies to opening a connection or sending a request to a back-end.

Grace 10 is how long Pound should continue to answer existing connections after a receiving and INT or HUP signal (default: 30 seconds). The configured listeners are closed immediately. You can bypass this behavior by stopping Pound with a TERM or QUIT signal, in which case the program exits without any delay. (Default: 30 seconds)

HELPFUL HINT: If you run OpenBSD or FreeBSD you may want to look at relayd as a reverse proxy. Check out our Relayd proxy "how to" (relayd.conf) Guide.

ListenHTTP An HTTP listener defines an address and port that Pound will listen on for HTTP requests. All configuration directives enclosed between ListenHTTP and End are specific to a single HTTP listener. At the very least you must specify and address and a port for each listener.

Address 127.0.0.1 and Port 8081 are the ip address and port pound is going to listen for external connections on.

CheckURL "(^\/|\.html|\.css|\.jpg|favicon\.ico|robots\.txt|\.png)$" matches the incoming request. If a request fails to match than this service will be skipped and next one tried. If all services fail to match Pound returns an error. The example URL string specifies the file types we expect a client to want to retrieve. If the client tries to get any file other than those listed the request will fail. The dollar sign ($) says that all the strings listed must be located at the end of the request URL. This line will allow:

HeadRemove "X-Forwarded-For" Remove certain headers from the incoming requests. All occurrences of the matching specified header will be removed. Some clients already have a "X-Forwarded-For" header as they may be using a proxy on their end. If the "X-Forwarded-For" already exists then Pound will add the clients ip to the end of the existing header. This causes multiple comma separated ip address to show up in the web server logs. This line removes the header to pound can insert a clean one.

MaxRequest 1024 is the maximum request size in bytes. The example does not expect to see any uploads (POST) so all requests (GET and HEAD only) should be less than 1024 bytes or 1 KB. If the request is any larger the connection is denied. We have seen abusive SEARCH requests exceed 65KB before. Note that:

xHTTP 0 Defines which HTTP verbs are accepted. 0 (default) accept only standard HTTP requests (GET, POST, HEAD). Clients use GET to retrieve files like *.html and *.jpgs, HEAD displays data about the server or page without downloading the whole page, and POST is used to upload files to the server. See the "Questions?" section at the bottom of this page for help limiting requests to GET and HEAD only.

Err414, Err500, Err501, and Err503 are the custom error pages we are going to send back to clients who are denied access tot he back end servers by pound. You can make a HTML or standard text file for the error pages. If you decide not to specify error page files then pound will substitute its own. If you do use your own, make sure they are less than 1.46 kilobytes (1460 bytes). At this size an error page can be sent back to the client in just one packet.

Service is a definition of which back-end servers Pound will use to reply to incoming requests. A service may be defined as part of a listener (in which case it will be used only by that listener), or globally (which makes it available to all listeners). Pound will always try the private services in the order defined, followed by the global ones. All configuration directives enclosed between Service and End are specific to a single service.

HeadRequire "(Host: your_host.com|Host: www.your_host.com)" says the request _MUST_ contain at least one header matching the given pattern. Multiple HeadRequire directives may be defined per service, in which case any of them must be satisfied. The example says that a client must specify the name of the URL "your_host.com" or "www.your_host.com" to be allowed access to the backend servers. If they try to use the ip address, any other text string or no "host" header they are denied.

BackEnd is a definition of a single back-end server Pound will use to reply to incoming requests. All configuration directives enclosed between BackEnd and End are specific to a single service.

Address 127.0.0.1 and Port 8080 are the destination ip address and port of the backend web server. This is where all requests that pass the Pound checks will be sent. We are only using one backend web server, but you can add more. Check out the man page for more examples.

Emergency will be used once all existing backends are "dead". The logs will say "connect_nb: connect failed: No route to host" when a backend is not reachable. When a backend machine comes back up the log will say something like, "BackEnd 127.0.0.1:8080 resurrect" and the emergency address will stop being used. All configuration directives enclosed between Emergency and End are specific to a single service. If you do not have an emergency server server then just comment out the lines.

Address 127.0.0.1 and Port 8888 are the destination ip address and port of the emergency web server. This is where all requests will go if _all_ of the backend servers are unreachable.

Testing Pound

To start pound manually execute the following line:

/usr/local/sbin/pound -f /etc/pound.conf

The logs of pound will go into /var/log/daemon .

Starting Pound on boot

You can put the following into your /etc/rc.local to start the Pound daemon on boot.

# pound
if [ -x /usr/local/sbin/pound ]; then
   echo -n ' pound'; /usr/local/sbin/pound -f /etc/pound.conf
fi

Want more speed? Make sure to also check out the Network Speed and Performance Guide. With a little time and understanding you could easily double your firewall's throughput.

Questions?

Can I have Pound only accept the request methods GET and HEAD, but _not_ POST ?If you do not accept uploads then there is no reason to accept the POST request method. You can edit the pound source code file "config.c" and remove the reference to POST. Then, when you use the directive "xHTTP 0" in the pound.conf file it will only accept GET and HEAD.

vi config.c

  old line #:86   "^(GET|POST|HEAD) ([^ ]+) HTTP/1.[01]$",
  new line #:86   "^(GET|HEAD) ([^ ]+) HTTP/1.[01]$",

Now, just re-compile Pound like in the example above.

What hardware can I run Pound on? How much traffic can Pound handle?On a single core amd64 2.4GHz with two Intel PRO/1000MT gigabit nics, Pound can handle 850 requests per second with only a load of 0.3 (15 minute average).

Why does Pound run two processes running on the same port? One process is for the daemon listening for external connections and the other is for the poundctl control interface. With poundctl you can command Pound to enable/disable backend hosts manually and show information about how many and how fast clients are connecting. If you would like to disable the command process and thus disable poundctl access then you need to build (configure) with the "--disable-super" directive like so:

make clean; ./configure --with-ssl=/usr/local/ssl/ --disable-super

Pound sometimes says "The service is not available. Please try again later." to the client even though there is no load on the backend web servers. What could be wrong? If you are receiving 500 requests per second or more you may be running out of open files or "file descriptors". Type "ulimit -a" to find out how many open files are allowed (-n) as the default is anywhere between 64 and 1024 depending on your OS. You can increase the amount of open files and thus the amount of client connections by using "ulimit -n ". For example, to allow pound to accept 5,000 connections and forward 5,000 connection to back end servers (10,000 total) use "ulimit -n 10000".

I built Pound, but I do not see the man pages (pound.8 and poundctl.8).The source code installs the man pages in /usr/local/share/man/man8/ where you may not have your man path set to look. You can always copy the pound man pages from /usr/local/share/man/man8/ to /usr/local/man/man8/ like so:

cp /usr/local/share/man/man8/pound* /usr/local/man/man8/