{"id":3118,"date":"2019-05-28T22:36:37","date_gmt":"2019-05-28T19:36:37","guid":{"rendered":"https:\/\/kifarunix.com\/?p=3118"},"modified":"2019-05-28T22:36:38","modified_gmt":"2019-05-28T19:36:38","slug":"monitor-squid-logs-with-grafana-and-graylog","status":"publish","type":"post","link":"https:\/\/kifarunix.com\/monitor-squid-logs-with-grafana-and-graylog\/","title":{"rendered":"Monitor Squid logs with Grafana and Graylog"},"content":{"rendered":"\n<p>In this guide, we are going to learn how to monitor squid logs with Grafana and Graylog. You can check our other guides on installing Graylog, forwarding squid logs to Graylog and creating Graylog squid log field extractors by following the links below;<\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/install-graylog-3-0-on-centos-7\/\" target=\"_blank\" rel=\"noreferrer noopener\" aria-label=\" (opens in a new tab)\">Install Graylog 3.0 on CentOS 7<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/monitor-squid-access-logs-with-graylog-server\/\" target=\"_blank\" rel=\"noreferrer noopener\" aria-label=\" (opens in a new tab)\">Monitor Squid Access Logs with Graylog Server<\/a><\/p>\n\n\n\n<p><a rel=\"noreferrer noopener\" aria-label=\" (opens in a new tab)\" href=\"https:\/\/kifarunix.com\/create-squid-logs-extractors-on-graylog-server\/\" target=\"_blank\">Create Squid Logs Extractors on Graylog Server<\/a><\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Monitor Squid logs with Grafana and Graylog<\/h2>\n\n\n\n<p>Grafana is an opensource tool for visualizing data collected from different types of data stores such as Prometheus, InfluxDB, Elasticsearch, Graphite, MySQL and several other databases. In this case of integrating it with Graylog, we will use Elasticsearch as our Grafana datasource.<\/p>\n\n\n\n<p>To learn how to install Grafana on Ubuntu, Debian or Fedora, see the links below;<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li><a rel=\"noreferrer noopener\" aria-label=\" (opens in a new tab)\" href=\"https:\/\/kifarunix.com\/install-grafana-monitoring-tool-on-fedora-29\/\" target=\"_blank\">Install Grafana Monitoring Tool on Fedora 29<\/a><\/li><li><a rel=\"noreferrer noopener\" aria-label=\" (opens in a new tab)\" href=\"https:\/\/kifarunix.com\/install-grafana-metrics-monitoring-tool-on-debian-9\/\" target=\"_blank\">Install Grafana Metrics Monitoring Tool on Debian 9<\/a><\/li><li><a href=\"https:\/\/kifarunix.com\/install-grafana-data-visualization-tool-on-ubuntu-18-04\/\" target=\"_blank\" rel=\"noreferrer noopener\" aria-label=\" (opens in a new tab)\">Install Grafana Data Visualization Tool on Ubuntu 18.04<\/a><\/li><\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Configure Elasticsearch Remote Connection<\/h3>\n\n\n\n<p>Elasticsearch is listening on localhost by default in Graylog server. To configure it to allow remote connection, you need define an interface IP for the <strong>network.host<\/strong> parameter.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>vim \/etc\/elasticsearch\/elasticsearch.yml<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>...\nnetwork.host: GRAYLOG_SERVER_IP\n...<\/code><\/pre>\n\n\n\n<p>If firewall is running, open the Elasticsearch ports<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>firewall-cmd --add-port=9200\/tcp --permanent\nfirewall-cmd --reload<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>ufw allow from <strong>Graylog_IP<\/strong> to any port 9200 proto tcp<\/code><\/pre>\n\n\n\n<p>Restart Elasticsearch<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>systemctl restart elasticsearch<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Verify Elasticsearch Connection<\/h3>\n\n\n\n<p>Login to Grafana and run the command below to verify connection to Elasticsearch by running the command below;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>curl http:\/\/Graylog_IP_)R_HOSTNAME:9200\n{\n  \"name\" : \"x55YNL_\",\n  \"cluster_name\" : \"graylog\",\n  \"cluster_uuid\" : \"CQBqPDoCRKW7tt955kq5Uw\",\n  \"version\" : {\n    \"number\" : \"6.8.0\",\n    \"build_flavor\" : \"default\",\n    \"build_type\" : \"rpm\",\n    \"build_hash\" : \"65b6179\",\n    \"build_date\" : \"2019-05-15T20:06:13.172855Z\",\n    \"build_snapshot\" : false,\n    \"lucene_version\" : \"7.7.0\",\n    \"minimum_wire_compatibility_version\" : \"5.6.0\",\n    \"minimum_index_compatibility_version\" : \"5.0.0\"\n  },\n  \"tagline\" : \"You Know, for Search\"\n}<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Create Graylog Squid Logs Elasticsearch Index Set<\/h3>\n\n\n\n<p>Graylog uses one or more sets of Elasticsearch indices to optimize search and analysis operations for speed and low resource consumption.<\/p>\n\n\n\n<p>To create an index, navigate to <strong>System &gt; Indices<\/strong>. Hit Create index set. On the index configuration page, set the name of the index, description, a unique&nbsp;prefix for use in Elasticsearch, number of Elasticsearch shards, index rotation strategy.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><a href=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2019\/05\/graylog-index-configuration.png\"><img loading=\"lazy\" decoding=\"async\" width=\"1327\" height=\"603\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2019\/05\/graylog-index-configuration.png\" alt=\"Graylog squid logs index\" class=\"wp-image-3120\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2019\/05\/graylog-index-configuration.png 1327w, https:\/\/kifarunix.com\/wp-content\/uploads\/2019\/05\/graylog-index-configuration-768x349.png 768w\" sizes=\"(max-width: 1327px) 100vw, 1327px\" \/><\/a><\/figure>\n\n\n\n<figure class=\"wp-block-image\"><a href=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2019\/05\/graylog-index_configuration.png\"><img loading=\"lazy\" decoding=\"async\" width=\"1327\" height=\"505\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2019\/05\/graylog-index_configuration.png\" alt=\"Graylog squid logs ES index\" class=\"wp-image-3121\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2019\/05\/graylog-index_configuration.png 1327w, https:\/\/kifarunix.com\/wp-content\/uploads\/2019\/05\/graylog-index_configuration-768x292.png 768w\" sizes=\"(max-width: 1327px) 100vw, 1327px\" \/><\/a><\/figure>\n\n\n\n<p>Once you are done, click <strong>save<\/strong> to save the index.<\/p>\n\n\n\n<p>To verify the index name for your Elasticsearch datasource;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>curl -XGET graylog.example.com:9200\/_cat\/indices?v\nhealth status index         uuid                   pri rep docs.count docs.deleted store.size pri.store.size\ngreen  open   <strong>squidaccess_0<\/strong> EiMgXL2UQqWym-5VZ-atDg   1   0       8859            0      1.9mb          1.9mb<\/code><\/pre>\n\n\n\n<p>Our index in this case is <strong>squidaccess_0<\/strong>. Note that using the Graylog Elasticsearch indices may bring issues due to constant rotation. We will look at the possible work around in our next guide.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Add Grafana Datasource<\/h3>\n\n\n\n<p>If connection to Elasticsearch from Grafana server is okay, proceed to create Grafana Elasticsearch datasource. To add Grafana datasource, navigate to <strong>Configuration &gt; Data Sources<\/strong>.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><a href=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2019\/05\/grafana-datasource.png\"><img loading=\"lazy\" decoding=\"async\" width=\"1189\" height=\"253\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2019\/05\/grafana-datasource.png\" alt=\"Add Grafana Datasource\" class=\"wp-image-3119\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2019\/05\/grafana-datasource.png 1189w, https:\/\/kifarunix.com\/wp-content\/uploads\/2019\/05\/grafana-datasource-768x163.png 768w\" sizes=\"(max-width: 1189px) 100vw, 1189px\" \/><\/a><\/figure>\n\n\n\n<p>Click <strong>Add data source<\/strong> and choose Elasticsearch. Under the Elasticsearch datasource settings, set the name of the datasource, the URL of the Graylog Elasticsearch, Elasticsearch index prefix as defined in Graylog index above, time field name (<strong>timestamp<\/strong>).<\/p>\n\n\n\n<figure class=\"wp-block-image\"><a href=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2019\/05\/grafana-ds-settings.png\"><img loading=\"lazy\" decoding=\"async\" width=\"864\" height=\"631\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2019\/05\/grafana-ES-ds-settings.png\" alt=\"Monitor Squid logs with Grafana and Graylog: Creating Grafana Elasticsearch datasource\" class=\"wp-image-3143\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2019\/05\/grafana-ES-ds-settings.png 864w, https:\/\/kifarunix.com\/wp-content\/uploads\/2019\/05\/grafana-ES-ds-settings-768x561.png 768w\" sizes=\"(max-width: 864px) 100vw, 864px\" \/><\/a><\/figure>\n\n\n\n<p>Next, click <strong>Save &amp; Test <\/strong>to test the connection to Elasticsearch datasource. If everything is fine, then you should get Index Ok.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><a href=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2019\/05\/grafana-es-index.png\"><img loading=\"lazy\" decoding=\"async\" width=\"954\" height=\"301\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2019\/05\/grafana-es-index.png\" alt=\"Grafana Elasticsearch index Ok\" class=\"wp-image-3124\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2019\/05\/grafana-es-index.png 954w, https:\/\/kifarunix.com\/wp-content\/uploads\/2019\/05\/grafana-es-index-768x242.png 768w\" sizes=\"(max-width: 954px) 100vw, 954px\" \/><\/a><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\">Create Grafana Dashboard for Squid Logs<\/h3>\n\n\n\n<p>Once you have you Graylog Elasticsearch datasource added to Grafana, you need to create the dashboards for visualizing the data. This involves creating various queries for different dashboards you may want to have. You can also import a ready made dashboard.<\/p>\n\n\n\n<p>To create a new or import Grafana dashboard, click on the <strong>HOME<\/strong> dropdown on the top left corner and choose whether to import dashboard json file or create a new one.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><a href=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2019\/05\/grafana-new-dashboard.png\"><img loading=\"lazy\" decoding=\"async\" width=\"913\" height=\"330\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2019\/05\/grafana-new-dashboard.png\" alt=\"New grafana dashboard\" class=\"wp-image-3145\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2019\/05\/grafana-new-dashboard.png 913w, https:\/\/kifarunix.com\/wp-content\/uploads\/2019\/05\/grafana-new-dashboard-768x278.png 768w\" sizes=\"(max-width: 913px) 100vw, 913px\" \/><\/a><\/figure>\n\n\n\n<p>For example, based on my Graylog squid log extractor, this is a simple dashboard that we have created.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><a href=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2019\/05\/squid-logs-dashboard.png\"><img loading=\"lazy\" decoding=\"async\" width=\"1351\" height=\"646\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2019\/05\/squid-logs-dashboard.png\" alt=\"Grafana-Graylog Squid access logs dashboard\" class=\"wp-image-3144\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2019\/05\/squid-logs-dashboard.png 1351w, https:\/\/kifarunix.com\/wp-content\/uploads\/2019\/05\/squid-logs-dashboard-768x367.png 768w\" sizes=\"(max-width: 1351px) 100vw, 1351px\" \/><\/a><\/figure>\n\n\n\n<p>Below are the panels that makes up this dashboard.<\/p>\n\n\n\n<p>Total Traffic:<\/p>\n\n\n\n<figure class=\"wp-block-image\"><a href=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2019\/05\/top-traffic-panel.png\"><img loading=\"lazy\" decoding=\"async\" width=\"1328\" height=\"500\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2019\/05\/top-traffic-panel.png\" alt=\"Grafana squid total traffic\" class=\"wp-image-3146\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2019\/05\/top-traffic-panel.png 1328w, https:\/\/kifarunix.com\/wp-content\/uploads\/2019\/05\/top-traffic-panel-768x289.png 768w\" sizes=\"(max-width: 1328px) 100vw, 1328px\" \/><\/a><\/figure>\n\n\n\n<p>Top Sites:<\/p>\n\n\n\n<figure class=\"wp-block-image\"><a href=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2019\/05\/top-sites.png\"><img loading=\"lazy\" decoding=\"async\" width=\"1342\" height=\"518\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2019\/05\/top-sites.png\" alt=\"top sites\" class=\"wp-image-3147\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2019\/05\/top-sites.png 1342w, https:\/\/kifarunix.com\/wp-content\/uploads\/2019\/05\/top-sites-768x296.png 768w\" sizes=\"(max-width: 1342px) 100vw, 1342px\" \/><\/a><\/figure>\n\n\n\n<p>Top 10 Denied Sites:<\/p>\n\n\n\n<figure class=\"wp-block-image\"><a href=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2019\/05\/top-10-denied.png\"><img loading=\"lazy\" decoding=\"async\" width=\"1332\" height=\"540\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2019\/05\/top-10-denied.png\" alt=\"top 10 denied\" class=\"wp-image-3148\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2019\/05\/top-10-denied.png 1332w, https:\/\/kifarunix.com\/wp-content\/uploads\/2019\/05\/top-10-denied-768x311.png 768w\" sizes=\"(max-width: 1332px) 100vw, 1332px\" \/><\/a><\/figure>\n\n\n\n<p>Top user:<\/p>\n\n\n\n<figure class=\"wp-block-image\"><a href=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2019\/05\/top-user.png\"><img loading=\"lazy\" decoding=\"async\" width=\"1325\" height=\"530\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2019\/05\/top-user.png\" alt=\"top user\" class=\"wp-image-3149\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2019\/05\/top-user.png 1325w, https:\/\/kifarunix.com\/wp-content\/uploads\/2019\/05\/top-user-768x307.png 768w\" sizes=\"(max-width: 1325px) 100vw, 1325px\" \/><\/a><\/figure>\n\n\n\n<p>That is just the basics on how to monitor squid logs with Grafana and Graylog. We hope this was informative enough. Feel free to drop your comments. Enjoy.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>In this guide, we are going to learn how to monitor squid logs with Grafana and Graylog. You can check our other guides on installing<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"rank_math_lock_modified_date":false,"footnotes":""},"categories":[301,962,121,72],"tags":[964,302,963,966],"class_list":["post-3118","post","type-post","status-publish","format-standard","hentry","category-grafana","category-graylog","category-howtos","category-monitoring","tag-elasticsearch","tag-grafana","tag-graylog","tag-squid-logs","generate-columns","tablet-grid-50","mobile-grid-100","grid-parent","grid-50"],"_links":{"self":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/3118"}],"collection":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/comments?post=3118"}],"version-history":[{"count":4,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/3118\/revisions"}],"predecessor-version":[{"id":3151,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/3118\/revisions\/3151"}],"wp:attachment":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media?parent=3118"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/categories?post=3118"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/tags?post=3118"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}