{"id":8484,"date":"2021-03-31T23:16:36","date_gmt":"2021-03-31T20:16:36","guid":{"rendered":"https:\/\/kifarunix.com\/?p=8484"},"modified":"2024-03-19T18:13:27","modified_gmt":"2024-03-19T15:13:27","slug":"forward-apache-logs-to-central-log-server-with-rsyslog","status":"publish","type":"post","link":"https:\/\/kifarunix.com\/forward-apache-logs-to-central-log-server-with-rsyslog\/","title":{"rendered":"Forward Apache Logs to Central Log Server with Rsyslog"},"content":{"rendered":"\n<p>In this tutorial, you will learn how to forward Apache logs to central log server with rsyslog.  Apache do not log to syslog by default.  So, what if you want to forward them logs to central system where you can easily manage them? In that case, there are various tools which can be used to read and sent Apache logs to central logging systems.<\/p>\n\n\n\n<p>Want to setup Central Log server with Rsyslog? Check the guides below;<\/p>\n\n\n\n<p><a aria-label=\" (opens in a new tab)\" href=\"https:\/\/kifarunix.com\/setup-rsyslog-server-on-debian-10\/\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"rank-math-link\">Setup Rsyslog Server on Debian 10<\/a><\/p>\n\n\n\n<p><a rel=\"noreferrer noopener\" class=\"rank-math-link\" href=\"https:\/\/kifarunix.com\/setup-rsyslog-server-on-ubuntu-20-04\/\" target=\"_blank\">Setup Rsyslog Server on Ubuntu 20.04<\/a><\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Forwarding Apache Logs to Central Log Server with Rsyslog<\/h2>\n\n\n\n<p>In this tutorial, we will explore two methods of forwarding Apache logs to central log server with rsyslog:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><a href=\"#imfile\" class=\"rank-math-link\">Monitoring Apache Log File with Rsyslog Text File Input Module<\/a><\/li>\n\n\n\n<li><a href=\"#logger\" class=\"rank-math-link\">Configure Apache to Log to Syslog<\/a><\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"imfile\">Monitoring Apache Log File with Rsyslog Text File Input Module<\/h3>\n\n\n\n<p>The rsyslog <code><a aria-label=\" (opens in a new tab)\" href=\"https:\/\/www.rsyslog.com\/doc\/v8-stable\/configuration\/modules\/imfile.html\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"rank-math-link\">text file input module (imfile)<\/a><\/code>, provides the ability to convert any standard text file into a syslog message.&nbsp;This module can read a log file line by line while passing each read line to rsyslog engine rules, which then <em>applies filter conditions and selects which actions needs to be carried out. Empty lines are&nbsp;<strong>not<\/strong>&nbsp;processed, as they would result in empty syslog records. They are simply ignored.<\/em><\/p>\n\n\n\n<p>In order to use Rsyslog text file input module to read Apache log file and forward it to remove Rsyslog server, you would create a configuration file like <code><strong>\/etc\/rsyslog.d\/02-apache2.conf<\/strong><\/code>, with the content below;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>vim \/etc\/rsyslog.d\/02-apache2.conf<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>module(load=\"imfile\" PollingInterval=\"10\" statefile.directory=\"\/var\/spool\/rsyslog\")\ninput(type=\"imfile\"\n      File=\"\/var\/log\/apache2\/error.log\"\n      Tag=\"http_error\"\n      Severity=\"error\"\n      Facility=\"local6\")\nlocal6.error        @192.168.59.12:514<\/code><\/pre>\n\n\n\n<p>Save and exit the file.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>The first line specifices the Rsyslog module to load, which in this case is the <code><strong>imfile<\/strong><\/code> module.<\/li>\n\n\n\n<li><code><strong>PollingInterval<\/strong><\/code>: Specifies how often the file is read for new data. You should never set the value of this parameter to 0 or you risk hogging your system CPU.<\/li>\n\n\n\n<li><code><strong>statefile.directory<\/strong><\/code>:  specify a dedicated directory for the storage of imfile state files. The file should  exist and be writable by rsyslog before restarting rsyslog.<br><code>ls -ald \/var\/spool\/rsyslog\/<\/code><br><code><strong>drwx------ 2 syslog adm 4096 Feb 11 2020 \/var\/spool\/rsyslog\/<\/strong><\/code><\/li>\n\n\n\n<li><code><strong>type<\/strong><\/code>: Specifies the type of the module.<\/li>\n\n\n\n<li><strong><code>File<\/code><\/strong>: Specifies the file to be polled.<\/li>\n\n\n\n<li><strong><code>Tag<\/code><\/strong>: a tag to be assigned to messages read from the file above.<\/li>\n\n\n\n<li><strong><code>Severity<\/code><\/strong>: syslog severity to be assigned to lines read from the file.<\/li>\n\n\n\n<li><strong><code>Facility<\/code><\/strong>: syslog facility to be assigned to messages read from the file specified.<\/li>\n\n\n\n<li>The last line specifies that the these log lines be forwarded to a remote server, <code><strong>192.168.59.12<\/strong><\/code>, on <code><strong>UDP<\/strong><\/code> port <code><strong>514<\/strong><\/code>.<\/li>\n<\/ul>\n\n\n\n<p>Verify the correctness of Rsyslog configuration file;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>rsyslogd -N1 -f \/etc\/rsyslog.d\/02-apache2.conf<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>rsyslogd: version 8.2006.0, config validation run (level 1), master config \/etc\/rsyslog.d\/02-apache2.conf\nrsyslogd: End of config validation run. Bye.<\/code><\/pre>\n\n\n\n<p>If no issues, restart Rsyslog;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>systemctl restart rsyslog<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\">Verify Reception of Apache Logs on Remote Log Server<\/h4>\n\n\n\n<p>You can use tcpdump to check if for any communication between the Rsyslog server and the client on UDP port 514; Replace the interface and the source host IP accordingly.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>tcpdump -i enp0s8 src host 192.168.59.13 and udp port 514 -nn -vv<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>tcpdump: listening on enp0s8, link-type EN10MB (Ethernet), capture size 262144 bytes\n14:44:27.229660 IP (tos 0x0, ttl 64, id 35360, offset 0, flags &#91;DF], proto UDP (17), length 1000)\n    192.168.59.13.35486 &gt; 192.168.59.12.514: &#91;udp sum ok] SYSLOG, length: 972\n\tFacility local6 (22), Severity error (3)\n\tMsg: Mar 31 21:44:27 ubuntu20 http_error &#91;Wed Mar 31 21:44:27.206708 2021] &#91;:error] &#91;pid 19242:tid 140596081583680] &#91;client 127.0.0.1:51706] ModSecurity: Warning. Matched \"Operator `PmFromFile' with parameter `unix-shell.data' against variable `ARGS:exec' (Value: `\/bin\/ls' ) &#91;file \"\/etc\/apache2\/modsecurity.d\/owasp-crs\/rules\/REQUEST-932-APPLICATION-ATTACK-RCE.conf\"] &#91;line \"496\"] &#91;id \"932160\"] &#91;rev \"\"] &#91;msg \"Remote Command Execution: Unix Shell Code Found\"] &#91;data \"Matched Data: bin\/ls found within ARGS:exec: \/bin\/ls\"] &#91;severity \"2\"] &#91;ver \"OWASP_CRS\/3.2.0\"] &#91;maturity \"0\"] &#91;accuracy \"0\"] &#91;tag \"application-multi\"] &#91;tag \"language-shell\"] &#91;tag \"platform-unix\"] &#91;tag \"attack-rce\"] &#91;tag \"paranoia-level\/1\"] &#91;tag \"OWASP_CRS\"] &#91;tag \"OWASP_CRS\/WEB_ATTACK\/COMMAND_INJECTION\"] &#91;tag \"WASCTC\/WASC-31\"] &#91;tag \"OWASP_TOP_10\/A1\"] &#91;tag \"PCI\/6.5.2\"] &#91;hostname \"127.0.1.1\"] &#91;uri \"\/\"] &#91;unique_id \"161721626740.953787\"] &#91;ref \"o1,6v11,7t:urlDecodeUni,t:cmdLine,t:normalizePath,t:lowercase\"]\n...<\/code><\/pre>\n\n\n\n<p>You can also check your remote log file on the server. In our setup, we configured our Rsyslog log server to log remote logs per the program sending the logs on per host directory.<\/p>\n\n\n\n<p>So in such a case, we have such logs from our Apache server;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>ls \/var\/log\/remotelogs\/192.168.59.13\/<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>http_error.log<\/code><\/pre>\n\n\n\n<p>Tailing this log file;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>tail -f \/var\/log\/remotelogs\/192.168.59.13\/http_error.log<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>2021-03-31T21:50:57-04:00 ubuntu20 http_error &#91;Wed Mar 31 21:50:57.617055 2021] &#91;:error] &#91;pid 19243:tid 140595980871232] &#91;client 192.168.59.1:43602] ModSecurity: Warning. Matched \"Operator `Rx' with parameter `^&#91;\\\\d.:]+$' against variable `REQUEST_HEADERS:Host' (Value: `192.168.59.13' ) &#91;file \"\/etc\/apache2\/modsecurity.d\/owasp-crs\/rules\/REQUEST-920-PROTOCOL-ENFORCEMENT.conf\"] &#91;line \"722\"] &#91;id \"920350\"] &#91;rev \"\"] &#91;msg \"Host header is a numeric IP address\"] &#91;data \"192.168.59.13\"] &#91;severity \"4\"] &#91;ver \"OWASP_CRS\/3.2.0\"] &#91;maturity \"0\"] &#91;accuracy \"0\"] &#91;tag \"application-multi\"] &#91;tag \"language-multi\"] &#91;tag \"platform-multi\"] &#91;tag \"attack-protocol\"] &#91;tag \"paranoia-level\/1\"] &#91;tag \"OWASP_CRS\"] &#91;tag \"OWASP_CRS\/PROTOCOL_VIOLATION\/IP_HOST\"] &#91;tag \"WASCTC\/WASC-21\"] &#91;tag \"OWASP_TOP_10\/A7\"] &#91;tag \"PCI\/6.5.10\"] &#91;hostname \"127.0.1.1\"] &#91;uri \"\/\"] &#91;unique_id \"161721665745.833955\"] &#91;ref \"o0,13v33,13\"]\n2021-03-31T21:50:57-04:00 ubuntu20 http_error &#91;Wed Mar 31 21:50:57.620882 2021] &#91;:error] &#91;pid 19243:tid 140595980871232] &#91;client 192.168.59.1:43602] ModSecurity: Warning. Matched \"Operator `PmFromFile' with parameter `unix-shell.data' against variable `ARGS:doc' (Value: `\/bin\/ls' ) &#91;file \"\/etc\/apache2\/modsecurity.d\/owasp-crs\/rules\/REQUEST-932-APPLICATION-ATTACK-RCE.conf\"] &#91;line \"496\"] &#91;id \"932160\"] &#91;rev \"\"] &#91;msg \"Remote Command Execution: Unix Shell Code Found\"] &#91;data \"Matched Data: bin\/ls found within ARGS:doc: \/bin\/ls\"] &#91;severity \"2\"] &#91;ver \"OWASP_CRS\/3.2.0\"] &#91;maturity \"0\"] &#91;accuracy \"0\"] &#91;tag \"application-multi\"] &#91;tag \"language-shell\"] &#91;tag \"platform-unix\"] &#91;tag \"attack-rce\"] &#91;tag \"paranoia-level\/1\"] &#91;tag \"OWASP_CRS\"] &#91;tag \"OWASP_CRS\/WEB_ATTACK\/COMMAND_INJECTION\"] &#91;tag \"WASCTC\/WASC-31\"] &#91;tag \"OWASP_TOP_10\/A1\"] &#91;tag \"PCI\/6.5.2\"] &#91;hostname \"127.0.1.1\"] &#91;uri \"\/\"] &#91;unique_id \"161721665745.833955\"] &#91;ref \"o1,6v10,7t:urlDecodeUni,t:cmdLine,t:normalizePath,t:lowercase\"]\n...<\/code><\/pre>\n\n\n\n<p>And that is how Rsyslog Text File Input Module can be used to read and forward application logs to a remote server.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"logger\">Configure Apache to Log to Syslog<\/h3>\n\n\n\n<p>In order to forward logs via Rsyslog, the application generating logs should be able to write to syslog. Apache doesnt write to syslog by default.<\/p>\n\n\n\n<p>In order to configure Apache to log to syslog, you can use a program like <code><strong>logger<\/strong><\/code>. Logger can be used to write application logs to syslog.<\/p>\n\n\n\n<p>By default, the <code>ErrorLog<\/code> and <code>CustomLog<\/code> directives to set the name of the file to which the server will log any errors it encounters and any request made to the server respectively.<\/p>\n\n\n\n<p>By default, these files are set like;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ErrorLog ${APACHE_LOG_DIR}\/error.log\nCustomLog ${APACHE_LOG_DIR}\/access.log combined<\/code><\/pre>\n\n\n\n<p>So error logs are logged to ${APACHE_LOG_DIR}\/error.log while access logs are logged to ${APACHE_LOG_DIR}\/access.log.<\/p>\n\n\n\n<p>Now that we want to forward Apache logs to remote syslog server you can set the logging directive to pipe the logs to logger which can then write to a specific syslog facility, specifying the appropriate severity.<\/p>\n\n\n\n<p>Let us take for example we want to log Apache error logs to syslog facility, local6 and severity we set to error i.e <code><strong>local6.err<\/strong><\/code>, edit the site configuration and set the value for the syslog directive as shown below;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>vim \/etc\/apache2\/sites-available\/000-default.conf<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>...\nErrorLog \"| \/usr\/bin\/logger -thttp_error: -plocal6.err\"\nCustomLog ${APACHE_LOG_DIR}\/access.log combined<\/code><\/pre>\n\n\n\n<p>Options passed to logger command; <strong><code>-t<\/code><\/strong> specifies the tag and <strong><code>-p<\/code><\/strong> specifies the syslog priority, <strong><code>facility.level<\/code><\/strong>.<\/p>\n\n\n\n<p>Save and exit the file.<\/p>\n\n\n\n<p>Next, configure Rsyslog to forward logs written to syslog priority, <code><strong>local6.err<\/strong><\/code> to a remote Rsyslog server.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>echo \"local6.err @192.168.59.12:514\" &gt;&gt; \/etc\/rsyslog.conf<\/code><\/pre>\n\n\n\n<p>If you want to log directly to remote server without specifying the server on the rsyslog.conf file, use the line;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>ErrorLog \"| \/usr\/bin\/logger -thttp_error: -plocal6.err -n 192.168.59.12 -P 514\"<\/code><\/pre>\n\n\n\n<p>Check Apache for any configuration errors;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>apachectl -t<\/code><\/pre>\n\n\n\n<p>If you get the output, <code><strong>Syntax OK<\/strong><\/code>, then you good to restart Apache;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>systemctl restart apache2<\/code><\/pre>\n\n\n\n<p>Similarly, you can check Rsyslog configurations for any errors;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>rsyslogd -N1<\/code><\/pre>\n\n\n\n<p>Restart Rsyslog;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>systemctl restart rsyslog<\/code><\/pre>\n\n\n\n<p>You should now be able to receive the logs at the remote central log server.<\/p>\n\n\n\n<p>One thing to note with this setting, <code><strong>ErrorLog \"| \/usr\/bin\/logger -thttp_error: -plocal6.err\"<\/strong><\/code>, it doesnt to logs the local system log file and hence, there is a high possibility that some logs may be lost.<\/p>\n\n\n\n<p>To overcome this and log the Apache logs to both a local file and a remote server, use the tee command to pipe logs to logger and to the local log file as shown below;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ErrorLog \"|\/bin\/sh -c '\/usr\/bin\/tee -a \/var\/log\/apache2\/error.log | \/usr\/bin\/logger -thttp_error: -plocal6.err'\"\nCustomLog ${APACHE_LOG_DIR}\/access.log combined<\/code><\/pre>\n\n\n\n<p>Configure Rsyslog to sent logs on priority local6.err to remote log server.<\/p>\n\n\n\n<p>Restart both Rsyslog and Apache;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>systemctl restart rsyslog apache2<\/code><\/pre>\n\n\n\n<p> You should now be able log to the local file and to remote log server.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Other Tutorials<\/h3>\n\n\n\n<p><a rel=\"noreferrer noopener\" href=\"https:\/\/kifarunix.com\/configure-rsyslog-on-solaris-11-4-to-send-logs-to-remote-log-server\/\" target=\"_blank\">Configure Rsyslog on Solaris 11.4 to Send logs to Remote Log Server<\/a><\/p>\n\n\n\n<p><a rel=\"noreferrer noopener\" href=\"https:\/\/kifarunix.com\/how-to-configure-syslog-to-send-logs-to-remote-syslog-server-on-solaris-11-4\/\" target=\"_blank\">Configure Syslog on Solaris 11.4 for Remote Logging<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/install-and-configure-nxlog-ce-on-ubuntu-20-04\/\" target=\"_blank\" aria-label=\" (opens in a new tab)\" rel=\"noreferrer noopener\" class=\"rank-math-link\">Install and Configure NXLog CE on Ubuntu 20.04<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>In this tutorial, you will learn how to forward Apache logs to central log server with rsyslog. Apache do not log to syslog by default.<\/p>\n","protected":false},"author":3,"featured_media":8508,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"rank_math_lock_modified_date":false,"footnotes":""},"categories":[191,121,72,331],"tags":[3371,3366,3372,3367,3368,3370,3369],"class_list":["post-8484","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-rsyslog","category-howtos","category-monitoring","category-syslog","tag-apache-central-log-server","tag-forward-apache-logs-to-remote-log-server-with-rsyslog","tag-forward-apache-logs-to-rsyslog-server","tag-forward-apache-logs-with-rsyslog","tag-log-apache-logs-to-rsyslog","tag-rsyslog-apache-logging","tag-rsyslog-forward-apache-logs","generate-columns","tablet-grid-50","mobile-grid-100","grid-parent","grid-50","resize-featured-image"],"_links":{"self":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/8484"}],"collection":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/comments?post=8484"}],"version-history":[{"count":9,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/8484\/revisions"}],"predecessor-version":[{"id":21867,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/8484\/revisions\/21867"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media\/8508"}],"wp:attachment":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media?parent=8484"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/categories?post=8484"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/tags?post=8484"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}