{"id":3083,"date":"2019-05-26T12:43:17","date_gmt":"2019-05-26T09:43:17","guid":{"rendered":"https:\/\/kifarunix.com\/?p=3083"},"modified":"2019-05-26T20:53:26","modified_gmt":"2019-05-26T17:53:26","slug":"monitor-squid-access-logs-with-graylog-server","status":"publish","type":"post","link":"https:\/\/kifarunix.com\/monitor-squid-access-logs-with-graylog-server\/","title":{"rendered":"Monitor Squid Access Logs with Graylog Server"},"content":{"rendered":"\n

Welcome to our tutorial on how to monitor squid access logs with Graylog server. Graylog<\/a> is a leading open-source log management tool that provides real time collection, storage, analysis and enrichment of machine data.<\/p>\n\n\n\n

Learn how to install Graylog 3 on CentOS 7 by following the link below.<\/p>\n\n\n\n

How to Install Graylog 3.0 Server on CentOS 7<\/a><\/p>\n\n\n\n

Monitor Squid Access Logs with Graylog server<\/h2>\n\n\n\n

Well, once you have setup your Graylog server, the next step would to ingest logs from various endpoints for analysis. This guide focuses on monitoring Pfsense squid proxy logs on Graylog. The Pfsense has been configured to forward Squid access logs to a central Rsyslog Server running on Ubuntu 18.04. You can as well forward Squid access logs directly to your Graylog server<\/p>\n\n\n\n

You can check our link below on how to configure Rsyslog as a central log server on Ubuntu 18.04.<\/p>\n\n\n\n

How to Configure Remote Logging with Rsyslog on Ubuntu 18.04<\/a><\/p>\n\n\n\n

Create Graylog Squid logs Input<\/h3>\n\n\n\n

In order to get your data to Graylog server, you need to configure Graylog message inputs to accept the data being sent from various endpoints. In this guide, we are going to configure Graylog to receive Squid data on a Syslog UDP port 5140. Be sure to use ports > 1024 to avoid permission issues with privileged ports (<1024).<\/p>\n\n\n\n

Therefore, login to your Graylog server and navigate to System > Inputs<\/strong>. On the Inputs configuration page, select the type of input, in this guide, Syslog UDP<\/strong> and click Launch new input<\/strong>.<\/p>\n\n\n\n

\"monitor<\/a><\/figure>\n\n\n\n

Syslog UDP input configuration wizard opens up. Select your Graylog node<\/strong>, Set the name of the input (Title<\/strong>), set the bind address<\/strong> (leave the default 0.0.0.0) and the port<\/strong> to listen on (5140 in this case). Then leave out other settings with defaults.<\/p>\n\n\n\n

\"Graylog<\/a><\/figure>\n\n\n\n

Once you are done, click the save<\/strong> button at the bottom of the configuration wizard to save the changes. If all is well, the Syslog UDP input will start to run immediately you save it.<\/p>\n\n\n\n

\"Graylog<\/a><\/figure>\n\n\n\n

Configure Remote Log Forwarding<\/h3>\n\n\n\n

Since Squid access logs are stored on a central Ubuntu 18.04 log server in this demo, we will go ahead and configure rsyslog to forward logs to Graylog server.<\/p>\n\n\n\n

Configure Rsyslog Forwarding<\/h4>\n\n\n\n

In our central log server, Squid access logs are stored under, \/var\/log\/remotelogs\/pfsense\/squid\/access.logs<\/strong>. Hence, to configure Rsyslog to forward these logs, create a new configuration file as shown below;<\/p>\n\n\n\n

vim \/etc\/rsyslog.d\/60-squid.conf<\/code><\/pre>\n\n\n\n
$ModLoad imfile\n$InputFileName \/var\/log\/remotelogs\/pfsense\/squid\/access.log\n$InputFileTag squid-access\n$InputFileStateFile stat-squid-access\n$InputFileSeverity info\n$InputFileFacility local7\n$InputRunFileMonitor\nlocal7.* @192.168.43.98:5140;RSYSLOG_SyslogProtocol23Format<\/code><\/pre>\n\n\n\n
Check Rsyslog for any misconfigurations;<\/p>\n\n\n\n
rsyslogd -N1<\/code><\/pre>\n\n\n\n
Restart Rsyslog<\/p>\n\n\n\n
systemctl restart rsyslog<\/code><\/pre>\n\n\n\n
To verify that the logs are getting to Graylog server, you can use tcpdump<\/strong> command.<\/p>\n\n\n\n
tcpdump -i enp0s8 src 192.168.43.142 and \"udp port 5140\"<\/code><\/pre>\n\n\n\n
View Squid Access Logs on Graylog server<\/h3>\n\n\n\n
To verify that logs are being received on the defined Graylog input, click show received messaged<\/strong> on your local input.<\/p>\n\n\n\n
\"Graylog<\/a><\/figure>\n\n\n\n
When you click on show received messages<\/strong>, you are redirected to Graylog search dashboard where you can view you messages.<\/p>\n\n\n\n
\"monitor<\/a><\/figure>\n\n\n\n
There you go. You got your Squid access logs on your Graylog server, However, the logs have not been parsed correctly simply because these logs do not conform rules defined in syslog RFCs. Therefore, you need to define Extractors<\/strong> to instruct Graylog nodes on how to extract data from any text in the received Squid log messages.<\/p>\n\n\n\n
See our next guide on how to create Graylog squid access logs extractor<\/a>.<\/p>\n\n\n\n
Otherwise, you can try the Content packs from Graylog Marketplace<\/a> to see if you can get any that works for your Squid logs.<\/p>\n\n\n\n
Other related monitoring guides;<\/p>\n\n\n\n
Nagios SNMP Monitoring of Linux Hosts on AlienVault USM\/OSSIM<\/a><\/p>\n\n\n\n
Configure Nagios Availability Monitoring on AlienVault USM\/OSSIM<\/a><\/p>\n\n\n\n
Install Elastic Stack 7 on Ubuntu 18.04\/Debian 9.8<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
Welcome to our tutorial on how to monitor squid access logs with Graylog server. Graylog is a leading open-source log management tool that provides real time<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"rank_math_lock_modified_date":false,"footnotes":""},"categories":[962,121,72],"tags":[963,251],"class_list":["post-3083","post","type-post","status-publish","format-standard","hentry","category-graylog","category-howtos","category-monitoring","tag-graylog","tag-squid","generate-columns","tablet-grid-50","mobile-grid-100","grid-parent","grid-50"],"_links":{"self":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/3083"}],"collection":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/comments?post=3083"}],"version-history":[{"count":7,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/3083\/revisions"}],"predecessor-version":[{"id":3117,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/3083\/revisions\/3117"}],"wp:attachment":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media?parent=3083"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/categories?post=3083"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/tags?post=3083"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}