{"id":7317,"date":"2020-11-29T13:37:44","date_gmt":"2020-11-29T10:37:44","guid":{"rendered":"https:\/\/kifarunix.com\/?p=7317"},"modified":"2024-03-14T23:37:11","modified_gmt":"2024-03-14T20:37:11","slug":"configure-elk-stack-alerting-with-elastalert","status":"publish","type":"post","link":"https:\/\/kifarunix.com\/configure-elk-stack-alerting-with-elastalert\/","title":{"rendered":"Configure ELK Stack Alerting with ElastAlert"},"content":{"rendered":"<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"994\" height=\"559\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2020\/11\/elk-alerting-with-elastalert-1.png\" alt=\"Configure ELK Stack Alerting with ElastAlert\" class=\"wp-image-13750\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2020\/11\/elk-alerting-with-elastalert-1.png?v=1661614222 994w, https:\/\/kifarunix.com\/wp-content\/uploads\/2020\/11\/elk-alerting-with-elastalert-1-768x432.png?v=1661614222 768w\" sizes=\"(max-width: 994px) 100vw, 994px\" \/><figcaption class=\"wp-element-caption\">ELK Stack alerting with ElastAlert<\/figcaption><\/figure><\/div>\n\n\n<p>Welcome to our tutorial on how to configure ELK Stack alerting with ElastAlert. As much as ELK Stack enables you to collect, process\/parse, index and visualize various system data, it can as well be configured to alert on various events. <em>The alerting features enable you to watch for changes or anomalies in your data and perform the necessary actions in response.if certains event conditions are met<\/em>. ELK stack supports alerting but it is available as a paid subscription and you need a license to use. A 30 day trial version is also available. Well, in this tutorial, we will be using the open-source alternative to Elasticsearch X-Pack alerting feature, ElastAlert.<\/p>\n\n\n\n<p><em><a href=\"https:\/\/elastalert.readthedocs.io\/en\/latest\/\" target=\"_blank\" aria-label=\"ElastAlert (opens in a new tab)\" rel=\"noreferrer noopener\" class=\"rank-math-link\">ElastAlert<\/a> is to be&nbsp;<a href=\"https:\/\/elastalert.readthedocs.io\/en\/latest\/elastalert.html#reliability\" target=\"_blank\" rel=\"noopener\">reliable<\/a>, highly&nbsp;<a href=\"https:\/\/elastalert.readthedocs.io\/en\/latest\/elastalert.html#modularity\" target=\"_blank\" rel=\"noopener\">modular<\/a>, and easy to&nbsp;<a href=\"https:\/\/elastalert.readthedocs.io\/en\/latest\/running_elastalert.html#tutorial\" target=\"_blank\" rel=\"noopener\">set up<\/a>&nbsp;and&nbsp;<a href=\"https:\/\/elastalert.readthedocs.io\/en\/latest\/elastalert.html#configuration\" target=\"_blank\" rel=\"noopener\">configure<\/a>.<\/em> <em>It works by combining Elasticsearch with two types of components, rule types and alerts. Elasticsearch is periodically queried and the data is passed to the rule type, which determines when a match is found. When a match occurs, it is given to one or more alerts, which take action based on the match. This is configured by a set of rules, each of which defines a query, a rule type, and a set of alerts.<\/em><\/p>\n\n\n\n<p><em>Several rule types with common monitoring paradigms are included with ElastAlert:<\/em><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><em>\u201cMatch where there are X events in Y time\u201d (<code>frequency<\/code>&nbsp;type)<\/em><\/li>\n\n\n\n<li><em>\u201cMatch when the rate of events increases or decreases\u201d (<code>spike<\/code>&nbsp;type)<\/em><\/li>\n\n\n\n<li><em>\u201cMatch when there are less than X events in Y time\u201d (<code>flatline<\/code>&nbsp;type)<\/em><\/li>\n\n\n\n<li><em>\u201cMatch when a certain field matches a blacklist\/whitelist\u201d (<code>blacklist<\/code>&nbsp;and&nbsp;<code>whitelist<\/code>&nbsp;type)<\/em><\/li>\n\n\n\n<li><em>\u201cMatch on any event matching a given filter\u201d (<code>any<\/code>&nbsp;type)<\/em><\/li>\n\n\n\n<li><em>\u201cMatch when a field has two different values within some time\u201d (<code>change<\/code>&nbsp;type)<\/em><\/li>\n<\/ul>\n\n\n\n<p>Currently, ElastAlert have built in support for these alert types:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><em>Command<\/em><\/li>\n\n\n\n<li><em>Email<\/em><\/li>\n\n\n\n<li><em>JIRA<\/em><\/li>\n\n\n\n<li><em>OpsGenie<\/em><\/li>\n\n\n\n<li><em>SNS<\/em><\/li>\n\n\n\n<li><em>HipChat<\/em><\/li>\n\n\n\n<li><em>Slack<\/em><\/li>\n\n\n\n<li><em>Telegram<\/em><\/li>\n\n\n\n<li><em>GoogleChat<\/em><\/li>\n\n\n\n<li><em>Debug<\/em><\/li>\n\n\n\n<li><em>Stomp<\/em><\/li>\n\n\n\n<li><em>theHive<\/em><\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Sending ELK Stack Alerts with ElastAlert<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"install-elastalert\"><a href=\"#install-elastalert\">Installing ElastAlert in Linux<\/a><\/h3>\n\n\n\n<p>There are quite a number of requirements for the installation of ElastAlert as outlined on the <a href=\"https:\/\/elastalert.readthedocs.io\/en\/latest\/running_elastalert.html#requirements\" target=\"_blank\" aria-label=\"requirements page (opens in a new tab)\" rel=\"noreferrer noopener\" class=\"rank-math-link\">requirements page<\/a>. These include;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Elasticsearch<\/li>\n\n\n\n<li>ISO8601 or Unix timestamped data<\/li>\n\n\n\n<li>Python 3.6<\/li>\n\n\n\n<li>pip, see requirements.txt<\/li>\n\n\n\n<li>Packages on Ubuntu 14.x: python-pip python-dev libffi-dev libssl-dev<\/li>\n<\/ul>\n\n\n\n<h4 class=\"wp-block-heading\">Install and Setup Elastic\/ELK Stack<\/h4>\n\n\n\n<p>Follow the links below to install and setup ELK\/Elastic Stack.<\/p>\n\n\n\n<p><a aria-label=\" (opens in a new tab)\" href=\"https:\/\/kifarunix.com\/install-elastic-elk-stack-on-ubuntu-20-04\/\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"rank-math-link\">Install ELK Stack on Ubuntu 20.04<\/a><\/p>\n\n\n\n<p><a aria-label=\" (opens in a new tab)\" href=\"https:\/\/kifarunix.com\/installing-elk-stack-on-centos-8\/\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"rank-math-link\">Installing ELK Stack on CentOS 8<\/a><\/p>\n\n\n\n<p><a aria-label=\" (opens in a new tab)\" href=\"https:\/\/kifarunix.com\/deploy-a-single-node-elastic-stack-cluster-on-docker-containers\/\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"rank-math-link\">Deploy a Single Node Elastic Stack Cluster on Docker Containers<\/a><\/p>\n\n\n\n<p><a aria-label=\" (opens in a new tab)\" href=\"https:\/\/kifarunix.com\/install-elastic-stack-7-on-fedora-30-fedora-29-centos-7\/\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"rank-math-link\">Install Elastic Stack 7 on Fedora 30\/Fedora 29\/CentOS 7<\/a><\/p>\n\n\n\n<p><a aria-label=\" (opens in a new tab)\" href=\"https:\/\/kifarunix.com\/install-elastic-stack-7-on-ubuntu-18-04-debian-9-8\/\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"rank-math-link\">Install Elastic Stack 7 on Ubuntu 18.04\/Debian 9.8<\/a><\/p>\n\n\n\n<p>Of course the log data collected are unix timestamped.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Installing Python 3 on Linux<\/h4>\n\n\n\n<p>In this demo, we are installing ElastAlert on our Elastic stack server running on a CentOS 8 system. Note that you can as well install Elastalert on the client from where you are shipping logs.<\/p>\n\n\n\n<p>As per the requirements above, Python 3.6 is needed for ElastAlert. On CentOS 8, you can install Python 3.6 by executing the command below (if not already installed);<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>dnf install python36 python3-devel<\/code><\/pre>\n\n\n\n<p>For other distros, refer to the respective documentation on installing Python 3.6.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Install PIP on Linux<\/h4>\n\n\n\n<p>Similarly, for our CentOS 8 system running Python 3, then you can install PIP by executing the command below;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>dnf install python3-pip<\/code><\/pre>\n\n\n\n<p> Refer to your OS distro for specifics on installing PIP.<\/p>\n\n\n\n<p>Similarly, you need to install GNU Compiler Collection (gcc);<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>dnf install gcc<\/code><\/pre>\n\n\n\n<p>You might need to install the <strong>Development Tools<\/strong>, which provides comprehensive build tools but installing GCC suffice.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Installing ElastAlert<\/h4>\n\n\n\n<p>Once the requirements of installing ElastAlert are in place, you can now install latest release version of ElastAlert.<\/p>\n\n\n\n<p>Well, you got two options here;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>You can install the latest released version of ElastAlert using pip:<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>pip3 install elastalert<\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>or you can simply clone the ElastAlert repository for the most recent changes:<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>git clone https:\/\/github.com\/Yelp\/elastalert.git \/opt\/elastalert<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>cd \/opt\/elastalert<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>pip3 install \"setuptools&gt;=11.3\" -U<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>python3 setup.py install<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\">Install Elasticsearch ElastAlert Module<\/h4>\n\n\n\n<p>Next, install ElastAlert Elasticsearch module (for ES version 5 and above).<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>pip3 install \"elasticsearch&gt;=5.0.0\"<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>cd ~<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Configuring ELK Stack Alerting with ElastAlert<\/h3>\n\n\n\n<p>You can now configure ElastAlert for ELK Stack alerting.<\/p>\n\n\n\n<p>First off, the ElastAlert (as per our installation method of cloning its Github repo) ships with example configuration file, <code><strong>\/opt\/elastalert\/config.yml.example<\/strong><\/code>.<\/p>\n\n\n\n<p>Rename this configuration file removing the <em>.example<\/em> suffix.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>cp \/opt\/elastalert\/config.yaml{.example,}<\/code><\/pre>\n\n\n\n<p>The configuration file is highly commented. By default, without comment and empty lines, this is how it looks like;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>rules_folder: example_rules\nrun_every:\n  minutes: 1\nbuffer_time:\n  minutes: 15\nes_host: elasticsearch.example.com\nes_port: 9200\nwriteback_index: elastalert_status\nwriteback_alias: elastalert_alerts\nalert_time_limit:\n  days: 2\n<\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li><code><strong>rules_folder<\/strong><\/code> is where ElastAlert will load rule configuration files from, which in our case is <strong><code>\/opt\/elastalert\/example_rules<\/code><\/strong>.<\/li>\n\n\n\n<li><strong><code>run_every<\/code>&nbsp;<\/strong>is how often ElastAlert will query Elasticsearch.<\/li>\n\n\n\n<li><code><strong>buffer_time<\/strong><\/code>&nbsp;is the size of the query window, stretching backwards from the time each query is run.<\/li>\n\n\n\n<li><code><strong>es_host<\/strong><\/code>&nbsp;is the address of an Elasticsearch cluster where ElastAlert will store data about its state, queries run, alerts, and errors. Each rule may also use a different Elasticsearch host to query against.<\/li>\n\n\n\n<li><code><strong>es_port<\/strong><\/code>&nbsp;is the port corresponding to&nbsp;<code>es_host<\/code>.<\/li>\n\n\n\n<li><code><strong>writeback_index<\/strong><\/code>&nbsp;is the name of the index in which ElastAlert will store data. We will create this index later.<\/li>\n\n\n\n<li><code><strong>alert_time_limit<\/strong><\/code>&nbsp;is the retry window for failed alerts.<\/li>\n<\/ul>\n\n\n\n<h4 class=\"wp-block-heading\">Define the Address and the Port for the Elasticsearch node;<\/h4>\n\n\n\n<p>Open the ElastAlert configuration file for editing;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>vim \/opt\/elastalert\/config.yaml<\/code><\/pre>\n\n\n\n<p>The only thing we gonna change in the default configuration file is the IP address and port for ES.<\/p>\n\n\n\n<p>Find the IP on which ES is listening<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>ss -altnp | grep :9200<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>LISTEN   0        128        &#91;::ffff:192.168.57.30]:9200                *:*<\/code><\/pre>\n\n\n\n<p>Then;<\/p>\n\n\n\n<pre class=\"scroll-sz\"><code>...\n# The Elasticsearch hostname for metadata writeback\n# Note that every rule can have its own Elasticsearch host\n<strong>es_host: 192.168.57.30\n<\/strong>\n# The Elasticsearch port\n<strong>es_port: 9200\n<\/strong>...\n<\/code><\/pre>\n\n\n\n<p>If your ES is configured with SSL\/Authentication, be sure to set the respective specifics on ElastAlert config file.<\/p>\n\n\n\n<p>Save and exit the config.<\/p>\n\n\n\n<p>This is how our config file then looks like;<\/p>\n\n\n\n<pre id=\"block-7c787244-b42f-4d5d-b4dc-5d17dc30a540\" class=\"wp-block-preformatted\">less \/opt\/elastalert\/config.yaml<\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>rules_folder: \/opt\/elastalert\/example_rules\nrun_every:\n  minutes: 1\nbuffer_time:\n  minutes: 15\nes_host: 192.168.57.30\nes_port: 9200\nwriteback_index: elastalert_status\nwriteback_alias: elastalert_alerts\nalert_time_limit:\n  days: 2\n<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\">Create ElastAlert Index on Elasticsearch<\/h4>\n\n\n\n<p>Create an ElastAlert index on Elasticsearch to enable it to store <em>information<\/em> <em>and metadata about its queries and alerts<\/em>.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>elastalert-create-index<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>Elastic Version: 7.8.1\nReading Elastic 6 index mappings:\nReading index mapping 'es_mappings\/6\/silence.json'\nReading index mapping 'es_mappings\/6\/elastalert_status.json'\nReading index mapping 'es_mappings\/6\/elastalert.json'\nReading index mapping 'es_mappings\/6\/past_elastalert.json'\nReading index mapping 'es_mappings\/6\/elastalert_error.json'\nNew index elastalert_status created\nDone!\n<\/code><\/pre>\n\n\n\n<p>If you encounter the error, <code><strong>AttributeError: module 'yaml' has no attribute 'FullLoader'<\/strong><\/code>, while creating the ElastAlert ES indices, you can  reinstall PyYAML;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>sudo pip3 install --ignore-installed PyYAML<\/code><\/pre>\n\n\n\n<p>After creating the indices, if you navigate to Kibana under stack management &gt; Elasticsearch &gt; Index management, you should be able to see such indices;<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1465\" height=\"584\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2020\/12\/elastalert-indices.png\" alt=\"\" class=\"wp-image-7322\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2020\/12\/elastalert-indices.png?v=1606770781 1465w, https:\/\/kifarunix.com\/wp-content\/uploads\/2020\/12\/elastalert-indices-768x306.png?v=1606770781 768w\" sizes=\"(max-width: 1465px) 100vw, 1465px\" \/><\/figure>\n\n\n\n<h4 class=\"wp-block-heading\">Creating ElastAlert Rules and Alerting<\/h4>\n\n\n\n<p>As per our setup, the ElastAlert rules are located under, <strong><code>\/opt\/elastalert\/example_rules<\/code><\/strong> directory.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>ls \/opt\/elastalert\/example_rules\/ -1<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>example_cardinality.yaml\nexample_change.yaml\nexample_frequency.yaml\nexample_new_term.yaml\nexample_opsgenie_frequency.yaml\nexample_percentage_match.yaml\nexample_single_metric_agg.yaml\nexample_spike_single_metric_agg.yaml\nexample_spike.yaml\njira_acct.txt\nssh-repeat-offender.yaml\nssh.yaml\n<\/code><\/pre>\n\n\n\n<p>ElastAlert supports different types of rules as explained on <a rel=\"noreferrer noopener\" href=\"https:\/\/elastalert.readthedocs.io\/en\/latest\/ruletypes.html#ruletypes\" target=\"_blank\">Rule Types<\/a> page as well various alert channel types as outlined on <a aria-label=\"ElastAlert Alerts (opens in a new tab)\" href=\"https:\/\/elastalert.readthedocs.io\/en\/latest\/ruletypes.html#alerts\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"rank-math-link\">ElastAlert Alerts<\/a> page.<\/p>\n\n\n\n<p>In this setup, we will test a few rule type and use email for alerting.<\/p>\n\n\n\n<h5 class=\"wp-block-heading\">ELK Stack Email Alerting on Multiple Failed SSH Logins<\/h5>\n\n\n\n<p>If you check under the example rules directory, we have SSH rules file, <code><strong>\/opt\/elastalert\/example_rules\/ssh.yaml<\/strong><\/code>.<\/p>\n\n\n\n<p>In this setup, we are collecting logs from the end points using Filebeat. We would be to alerted via mail in case there is a more than failed 5 login attempts on an end point. Hence, below is our sample SSH configuration without comment lines;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>vim \/opt\/elastalert\/example_rules\/ssh.yaml<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>name: Sample SSH Rule\ntype: frequency\nnum_events: 3\ntimeframe:\n  minutes: 1\nfilter:\n- query:\n    query_string:\n      query: \"event.type:authentication_failure\"\nindex: filebeat-*\nrealert:\n  minutes: 1\nquery_key:\n  - source.ip\ninclude:\n  - host.hostname\n  - user.name\n  - source.ip\ninclude_match_in_root: true\nalert_subject: \"SSH Bruteforce Attacks Detected on {}\"\nalert_subject_args:\n  - host.hostname\nalert_text: |-\n  Multiple SSH failed logins detected on {}.\n  Details of the event:\n          - User: {}\n          - Source IP: {}\nalert_text_args:\n  - host.hostname\n  - user.name\n  - source.ip\nalert:\n  - email:\n     from_addr: \"elk@kifarunix-demo.com\"\n     email: \"gentoo@kifarunix-demo.com\"\nalert_text_type: alert_text_only\n<\/code><\/pre>\n\n\n\n<p>The above search for the authentication failure event type on the index, filebeat-*. It then sent alerts if three failed login attempts are noticed in under a minute. Be sure to set the correct index name and the correct search string for your events.<\/p>\n\n\n\n<p>Save and exit the file once you are done making changes.<\/p>\n\n\n\n<p>Note, for email alerting to work, you need to configure SMTP for email relay. You can check the guide below for setting up Postfix on Ubuntu\/Fedora.<\/p>\n\n\n\n<p><a aria-label=\" (opens in a new tab)\" href=\"https:\/\/kifarunix.com\/configure-postfix-to-use-gmail-smtp-on-ubuntu-20-04\/\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"rank-math-link\">Configure Postfix to Use Gmail SMTP on Ubuntu 20.04<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/configure-postfix-to-use-gmail-smtp-on-ubuntu-18-04\/\" target=\"_blank\" aria-label=\" (opens in a new tab)\" rel=\"noreferrer noopener\" class=\"rank-math-link\">Configure Postfix to Use Gmail SMTP on Ubuntu 18.04<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/configure-postfix-as-send-only-smtp-server-on-fedora-29\/\" target=\"_blank\" aria-label=\" (opens in a new tab)\" rel=\"noreferrer noopener\" class=\"rank-math-link\">Configure Postfix as Send-Only SMTP Server on Fedora 29<\/a><\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Testing ElastAlert Rule<\/h4>\n\n\n\n<p>Once you have configured your rule, you need to test whether it actually works. ElastAlert provides a script called <code><strong>elastalert-test-rule<\/strong><\/code> for validating the configured rules.<\/p>\n\n\n\n<p>The script is installed under, \/usr\/local\/bin;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>which elastalert-test-rule<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>\/usr\/local\/bin\/elastalert-test-rule<\/code><\/pre>\n\n\n\n<p>For example, to test the SSH rule above, navigate to<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>elastalert-test-rule --config \/opt\/elastalert\/config.yaml \/opt\/elastalert\/example_rules\/ssh.yaml<\/code><\/pre>\n\n\n\n<p>The script shows the output similar to below;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>INFO:elastalert:Note: In debug mode, alerts will be logged to console but NOT actually sent.\n            To send them but remain verbose, use --verbose instead.\nDidn't get any results.\nINFO:elastalert:Note: In debug mode, alerts will be logged to console but NOT actually sent.\n                To send them but remain verbose, use --verbose instead.\n1 rules loaded\nINFO:apscheduler.scheduler:Adding job tentatively -- it will be properly scheduled when the scheduler starts\nINFO:elastalert:Queried rule SSH abuse (ElastAlert 3.0.1) - 2 from 2020-12-01 21:13 EAT to 2020-12-01 21:14 EAT: 0 \/ 0 hits\n\nWould have written the following documents to writeback index (default is elastalert_status):\n\nelastalert_status - {'rule_name': 'SSH abuse (ElastAlert 3.0.1) - 2', 'endtime': datetime.datetime(2020, 12, 1, 18, 14, 56, 92899, tzinfo=tzutc()), 'starttime': datetime.datetime(2020, 12, 1, 18, 13, 55, 492899, tzinfo=tzutc()), 'matches': 0, 'hits': 0, '@timestamp': datetime.datetime(2020, 12, 1, 18, 14, 56, 416136, tzinfo=tzutc()), 'time_taken': 0.08273959159851074}\n<\/code><\/pre>\n\n\n\n<p>As you can see, there SSH failed events currently match in our Filebeat index, <code><strong>INFO:elastalert:Queried rule SSH abuse (ElastAlert 3.0.1) - 2 from 2020-12-01 21:13 EAT to 2020-12-01 21:14 EAT: 0 \/ 0 hits<\/strong><\/code>.<\/p>\n\n\n\n<p>I am gonna simulate multiple failed ssh events and rerun the script.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>elastalert-test-rule --config \/opt\/elastalert\/config.yaml \/opt\/elastalert\/example_rules\/ssh.yaml<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>INFO:elastalert:Note: In debug mode, alerts will be logged to console but NOT actually sent.\n            To send them but remain verbose, use --verbose instead.\nDidn't get any results.\nINFO:elastalert:Note: In debug mode, alerts will be logged to console but NOT actually sent.\n                To send them but remain verbose, use --verbose instead.\n1 rules loaded\nINFO:apscheduler.scheduler:Adding job tentatively -- it will be properly scheduled when the scheduler starts\nINFO:elastalert:Queried rule Sample SSH Rule from 2020-12-01 22:04 EAT to 2020-12-01 22:05 EAT: 5 \/ 5 hits\nINFO:elastalert:Alert for Sample SSH Rule at 2020-12-01T22:05:06+03:00:\nINFO:elastalert:Multiple SSH failed logins detected on solr.\nDetails of the event:\n        - User: gen_t00\n        - Source IP: 192.168.57.1\n\n\n\nWould have written the following documents to writeback index (default is elastalert_status):\n\nsilence - {'exponent': 0, 'rule_name': 'Sample SSH Rule.192.168.57.1', '@timestamp': datetime.datetime(2020, 12, 1, 19, 5, 17, 14585, tzinfo=tzutc()), 'until': datetime.datetime(2020, 12, 1, 19, 6, 17, 14577, tzinfo=tzutc())}\n\nelastalert_status - {'rule_name': 'Sample SSH Rule', 'endtime': datetime.datetime(2020, 12, 1, 19, 5, 16, 979897, tzinfo=tzutc()), 'starttime': datetime.datetime(2020, 12, 1, 19, 4, 16, 379897, tzinfo=tzutc()), 'matches': 1, 'hits': 5, '@timestamp': datetime.datetime(2020, 12, 1, 19, 5, 17, 15185, tzinfo=tzutc()), 'time_taken': 0.012313127517700195}\n<\/code><\/pre>\n\n\n\n<p>As you can see above, we have 5 events in under one minute;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>INFO:elastalert:Queried rule Sample SSH Rule from 2020-12-01 21:30 EAT to 2020-12-01 21:31 EAT: 5 \/ 5 hits<\/code><\/pre>\n\n\n\n<p>Event Details (Email Body):<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Multiple SSH failed logins detected on solr.\nDetails of the event:\n        - User: gen_t00\n        - Source IP: 192.168.57.1<\/code><\/pre>\n\n\n\n<p>Use &#8211;help option to see the script arguments you can use. <\/p>\n\n\n\n<pre id=\"block-38438733-1d67-4154-a75e-c7ff8e6ff0f0\" class=\"wp-block-preformatted\">elastalert-test-rule --help<\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Running ElastAlert<\/h3>\n\n\n\n<p>Once you have confirmed that your query is working fine, it is time to run ElastAlert. ElastAlert can be run as a daemon via supervisord or via Python.<\/p>\n\n\n\n<p>You can as well run it on standard output using the elastalert binary, <code><strong>\/usr\/local\/bin\/elastalert<\/strong><\/code>.<\/p>\n\n\n\n<p>For example, you run ElastAlert against all rules defined in the rules directory;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>\/usr\/local\/bin\/elastalert --verbose --config \/opt\/elastalert\/config.yaml<\/code><\/pre>\n\n\n\n<p>To specify a specific rules file;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>\/usr\/local\/bin\/elastalert --verbose --config \/opt\/elastalert\/config.yaml --rule \/path\/to\/rules-file.yaml<\/code><\/pre>\n\n\n\n<p>E.g<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>\/usr\/local\/bin\/elastalert --verbose --config \/opt\/elastalert\/config.yaml --rule \/opt\/elastalert\/example_rules\/ssh.yaml<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>1 rules loaded\nINFO:elastalert:Starting up\nINFO:elastalert:Disabled rules are: []\nINFO:elastalert:Sleeping for 59.99993 seconds\nINFO:elastalert:Queried rule Sample SSH Rule from 2020-12-01 22:01 EAT to 2020-12-01 22:16 EAT: 8 \/ 8 hits\nINFO:elastalert:Queried rule Sample SSH Rule from 2020-12-01 22:16 EAT to 2020-12-01 22:31 EAT: 0 \/ 0 hits\nINFO:elastalert:Queried rule Sample SSH Rule from 2020-12-01 22:31 EAT to 2020-12-01 22:46 EAT: 0 \/ 0 hits\nINFO:elastalert:Queried rule Sample SSH Rule from 2020-12-01 22:46 EAT to 2020-12-01 23:01 EAT: 0 \/ 0 hits\nINFO:elastalert:Queried rule Sample SSH Rule from 2020-12-01 23:01 EAT to 2020-12-01 23:12 EAT: 0 \/ 0 hits\nINFO:elastalert:Sent email to ['gentoo@kifarunix-demo.com']\nINFO:elastalert:Ignoring match for silenced rule Sample SSH Rule.192.168.57.1\nINFO:elastalert:Ran Sample SSH Rule from 2020-12-01 22:01 EAT to 2020-12-01 23:12 EAT: 0 query hits (0 already seen), 2 matches, 1 alerts sent\n<\/code><\/pre>\n\n\n\n<p id=\"elastalert-systemd-service\"><a href=\"#elastalert-systemd-service\">In this setup, we run ElastAlert as a service;<\/a><\/p>\n\n\n\n<pre class=\"scroll-box\"><code>cat &gt; \/etc\/systemd\/system\/elastalert.service &lt;&lt; 'EOL'\n[Unit]\nDescription=ELK Stack ElastAlert Service\nAfter=elasticsearch.service\n \n[Service]\nType=simple\nWorkingDirectory=\/opt\/elastalert\nExecStart=\/usr\/local\/bin\/elastalert --verbose --config \/opt\/elastalert\/config.yaml\n \n[Install]\nWantedBy=multi-user.target\nEOL\n<\/code><\/pre>\n\n\n\n<p>Reload systemd configurations;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>systemctl daemon-reload<\/code><\/pre>\n\n\n\n<p>Start and enable the service to run on  boot;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>systemctl enable --now elastalert<\/code><\/pre>\n\n\n\n<p>Checking the status;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>systemctl status elastalert<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>\u25cf elastalert.service - ELK Stack ElastAlert Service\n   Loaded: loaded (\/etc\/systemd\/system\/elastalert.service; enabled; vendor preset: disabled)\n   Active: active (running) since Tue 2020-12-01 23:18:47 EAT; 38s ago\n Main PID: 7340 (elastalert)\n    Tasks: 12 (limit: 17931)\n   Memory: 43.9M\n   CGroup: \/system.slice\/elastalert.service\n           \u2514\u25007340 \/bin\/python3 \/usr\/local\/bin\/elastalert --verbose --config \/opt\/elastalert\/config.yaml\n\nDec 01 23:18:59 elastic.kifarunix-demo.com elastalert[7340]: INFO:elastalert:Queried rule SSH abuse - reapeat offender from 2020-12-01 22:31 EAT to 2020-12-01 22:46 EAT: 0&gt;\nDec 01 23:18:59 elastic.kifarunix-demo.com elastalert[7340]: INFO:elastalert:Queried rule Event spike from 2020-12-01 23:16 EAT to 2020-12-01 23:18 EAT: 0 \/ 0 hits\n...\n<\/code><\/pre>\n\n\n\n<p>To run with ElastAlert Python, see <a aria-label=\"running ElastAlert (opens in a new tab)\" href=\"https:\/\/elastalert.readthedocs.io\/en\/latest\/running_elastalert.html#running-elastalert\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"rank-math-link\">running ElastAlert<\/a>.<\/p>\n\n\n\n<p>Simulate the events and verify if any alert is send  and received on mail;<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"924\" height=\"457\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2020\/12\/elastalert-alerts.png\" alt=\"\" class=\"wp-image-7335\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2020\/12\/elastalert-alerts.png?v=1606854645 924w, https:\/\/kifarunix.com\/wp-content\/uploads\/2020\/12\/elastalert-alerts-768x380.png?v=1606854645 768w\" sizes=\"(max-width: 924px) 100vw, 924px\" \/><\/figure><\/div>\n\n\n<p>And that marks the end of our tutorial on how to send ELK stack alerts with ElastAlert via Email. Feel free to explore other alert channels.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Reference<\/h3>\n\n\n\n<p><a aria-label=\"Running ElastAlert for the First time (opens in a new tab)\" href=\"https:\/\/elastalert.readthedocs.io\/en\/latest\/running_elastalert.html\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"rank-math-link\">Running ElastAlert for the First time<\/a><\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Other Tutorials<\/h3>\n\n\n\n<p><a aria-label=\" (opens in a new tab)\" href=\"https:\/\/kifarunix.com\/monitor-linux-system-metrics-with-elk-stack\/\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"rank-math-link\">Monitor Linux System Metrics with ELK Stack<\/a><\/p>\n\n\n\n<p><a aria-label=\" (opens in a new tab)\" href=\"https:\/\/kifarunix.com\/visualize-wordpress-user-activity-logs-on-elk-stack\/\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"rank-math-link\">Visualize WordPress User Activity Logs on ELK Stack<\/a><\/p>\n\n\n\n<p><a aria-label=\" (opens in a new tab)\" href=\"https:\/\/kifarunix.com\/process-and-visualize-modsecurity-logs-on-elk-stack\/\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"rank-math-link\">Process and Visualize ModSecurity Logs on ELK Stack<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Welcome to our tutorial on how to configure ELK Stack alerting with ElastAlert. As much as ELK Stack enables you to collect, process\/parse, index and<\/p>\n","protected":false},"author":3,"featured_media":13748,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"rank_math_lock_modified_date":false,"footnotes":""},"categories":[910,121,72],"tags":[2919,2918,2922,2923,2921,2920,2917],"class_list":["post-7317","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-elastic-stack","category-howtos","category-monitoring","tag-configure-elasticsearch-alerts-with-elastalert","tag-configure-elk-stack-alerting-with-elastalert","tag-elastalert","tag-elastalert-email","tag-elk-alerts","tag-elk-email-alerting","tag-install-elastalert-on-linux","generate-columns","tablet-grid-50","mobile-grid-100","grid-parent","grid-50","resize-featured-image"],"_links":{"self":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/7317"}],"collection":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/comments?post=7317"}],"version-history":[{"count":18,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/7317\/revisions"}],"predecessor-version":[{"id":21548,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/7317\/revisions\/21548"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media\/13748"}],"wp:attachment":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media?parent=7317"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/categories?post=7317"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/tags?post=7317"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}