{"id":8589,"date":"2021-04-11T00:14:58","date_gmt":"2021-04-10T21:14:58","guid":{"rendered":"https:\/\/kifarunix.com\/?p=8589"},"modified":"2024-03-19T08:21:36","modified_gmt":"2024-03-19T05:21:36","slug":"configure-prometheus-email-alerting-with-alertmanager","status":"publish","type":"post","link":"https:\/\/kifarunix.com\/configure-prometheus-email-alerting-with-alertmanager\/","title":{"rendered":"Configure Prometheus Email Alerting with AlertManager"},"content":{"rendered":"\n

In this tutorial, you will learn how to configure Prometheus Email alerting with AlertManager. AlertManager<\/a> is used to handle alerts sent by client applications such as the Prometheus server. It takes care of deduplicating, grouping, and routing them to the correct receiver integration such as email, PagerDuty, or OpsGenie. It also takes care of silencing and inhibition of alerts<\/em>.<\/p>\n\n\n\n

There is more to Prometheus! Check the link below;<\/p>\n\n\n\n

Prometheus: Up & Running: Infrastructure and Application Performance Monitoring<\/a><\/p>\n\n\n\n

Configuring Prometheus Email Alerting with AlertManager<\/h2>\n\n\n\n

In our previous guide, we learnt how to monitor SSL\/TLS certificate expiry using Prometheus and Grafana<\/a>.<\/p>\n\n\n\n

We integrated Telegraf with Prometheus for SSL\/TLS certificate.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

From the above screenshot, you can see that you have 2591897<\/code><\/strong> seconds, which is equivalent to ~30 days (a month)<\/code><\/strong> before the certificate expires.<\/p>\n\n\n\n

With that duration of time, one might end up forgetting that SSL\/TLS certificates expires and hence may forget to renew the certificate. That is why it is important to configure Prometheus Email alerting such that, when a few days are due for certificate renewal, you can be notified via email.<\/p>\n\n\n\n

So, assuming that you already have Prometheus up and running, how can you use AlertMnager to configure Prometheus Email alerting?<\/p>\n\n\n\n

Therefore, open Prometheus configuration file and set alertmanager configurations, the Promtheus alert rules files as shown below;<\/p>\n\n\n\n

vim \/etc\/prometheus\/prometheus.yml<\/code><\/pre>\n\n\n\n
Update the lines below accordingly;<\/p>\n\n\n\n
# Alertmanager configuration\nalerting:\n  alertmanagers:\n  - static_configs:\n    - targets:\n       - localhost:9093\n\n# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.\nrule_files:\n   - \"alert_rules.yml\"<\/code><\/pre>\n\n\n\n
Without comment lines, this is how our Prometheus configuration file looks like;<\/p>\n\n\n\n
global:\n  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.\n  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.\nalerting:\n  alertmanagers:\n  - static_configs:\n    - targets:\n       - localhost:9093\nrule_files:\n   - \"alert_rules.yml\"<\/strong>\nscrape_configs:\n  - job_name: 'prometheus'\n    static_configs:\n    - targets: ['localhost:9090']\n  - job_name: 'SSL\/TLS Certs'\n    static_configs:\n    - targets: ['192.168.59.12:9273']<\/code><\/pre>\n\n\n\n
Save and exit the configuration file.<\/p>\n\n\n\n
Create Prometheus Alert Rules<\/h4>\n\n\n\n
“Alerting rules allow you to define alert conditions based on Prometheus expression language expressions and to send notifications about firing alerts to an external service. Whenever the alert expression results in one or more vector elements at a given point in time, the alert counts as active for these elements’ label sets<\/em>“.<\/p>\n\n\n\n
In our configuration file, we defined the Prometheus rules file as alert_rules.yml<\/strong><\/code>. This file should reside within the Prometheus configurations directory, \/etc\/prometheus<\/strong><\/code>.<\/p>\n\n\n\n
Hence, create the rules file with the content similar to below;<\/p>\n\n\n\n
vim \/etc\/prometheus\/alert_rules.yml<\/code><\/pre>\n\n\n\n
Note that in this example, we will be creating rule to alert us when the SSL\/TLS certificate is due to expire in a few days.<\/p>\n\n\n\n
As shown above, we have a certificate that is due for renewal in the next 30 days. To make this demo easy, we will create a rule to alert when have 30 or less days to certificate renewal.<\/p>\n\n\n\n
groups:\n- name: alert_rules\n  rules:\n  - alert: SSL_TLS_Cert_Expiry\n    expr: x509_cert_expiry{job=\"SSL\/TLS Certs\"} <= 2592000 \n    for: 1m\n    labels:\n      severity: critical\n    annotations:\n      summary: SSL\/TLS Certificate Expiry<\/code><\/pre>\n\n\n\n
Various setting values have been explained here<\/a>.<\/p>\n\n\n\n
Save and exit the file.<\/p>\n\n\n\n
Check if the rule files are valid or not;<\/p>\n\n\n\n
promtool check rules \/etc\/prometheus\/alert_rules.yml<\/code><\/pre>\n\n\n\n
Checking \/etc\/prometheus\/alert_rules.yml\n  SUCCESS: 1 rules found<\/code><\/pre>\n\n\n\n
If you check on Prometheus web interface Rules page;<\/p>\n\n\n\n
\"Configure<\/figure>\n\n\n\n
Install and Configure AlertManager<\/h3>\n\n\n\n
Install AlertManager<\/h4>\n\n\n\n
AlertManager can be installed using pre-compiled binaries that can be downloaded from Prometheus downloads section<\/a>.<\/p>\n\n\n\n
Hence, before you can install AlertManager;<\/p>\n\n\n\n
Create AlertManager system user and group as shown below;<\/p>\n\n\n\n
useradd -M -r -s \/bin\/false alertmanager<\/code><\/pre>\n\n\n\n
Next, navigate to the downloads section and grab the latest version of AlertManager. You simply use wget<\/strong> to download it. The current release version as of this writing is 0.21.0.<\/p>\n\n\n\n
VER=0.21.0<\/code><\/pre>\n\n\n\n
wget https:\/\/github.com\/prometheus\/alertmanager\/releases\/download\/v$VER\/alertmanager-$VER.linux-amd64.tar.gz<\/code><\/pre>\n\n\n\n
Extract the downloaded binary;<\/p>\n\n\n\n
tar xzf alertmanager-0.21.0.linux-amd64.tar.gz<\/code><\/pre>\n\n\n\n
Copy the AlertManager binary files, alertmanager<\/strong><\/code> and amtool<\/strong><\/code> to binary directory like \/usr\/local\/bin\/<\/strong><\/code>.<\/p>\n\n\n\n
cp alertmanager-0.21.0.linux-amd64\/{alertmanager,amtool} \/usr\/local\/bin\/<\/code><\/pre>\n\n\n\n
Next, create a configuration directory for AlertManager and copy the YAML configuration to that directory;<\/p>\n\n\n\n
cp alertmanager-0.21.0.linux-amd64\/alertmanager.yml \/etc\/alertmanager\/<\/code><\/pre>\n\n\n\n
Set the ownership of the AlertManager configuration directory and the binaries to alertmanager user created above;<\/p>\n\n\n\n
chown alertmanager: \/etc\/alertmanager\/alertmanager.yml \/usr\/local\/bin\/{alertmanager,amtool}<\/code><\/pre>\n\n\n\n
Configure AlertManager<\/h4>\n\n\n\n
In this setup, we will be sending the alerts via Email and we will use Gmail relay in that case.<\/p>\n\n\n\n
Hence, configure AlertManager as follows;<\/p>\n\n\n\n
vim \/etc\/alertmanager\/alertmanager.yml<\/code><\/pre>\n\n\n\n
global:\n  resolve_timeout: 5m\n\nroute:\n  group_by: ['alertname']\n  group_wait: 10s\n  group_interval: 10s\n  repeat_interval: 24h\n  receiver: 'email'\nreceivers:\n- name: 'email'\n  email_configs:\n  - to: 'recipient@gmail.com'\n    from: 'email_id@gmail.com'\n    smarthost: smtp.gmail.com:587\n    auth_username: 'email_id@gmail.com'\n    auth_identity: 'email_id@gmail.com'\n    auth_password: 'password'<\/code><\/pre>\n\n\n\n
Save and exit the configuration file. Be sure to set the email settings appropriately.<\/p>\n\n\n\n
More about the config on configuration<\/a> page and git repo<\/a> page.<\/p>\n\n\n\n
Check the Alertmanager configuration file to validate it.<\/p>\n\n\n\n
amtool check-config \/etc\/alertmanager\/alertmanager.yml<\/code><\/pre>\n\n\n\n
Checking '\/etc\/alertmanager\/alertmanager.yml'  SUCCESS\nFound:\n - global config\n - route\n - 0 inhibit rules\n - 1 receivers\n - 0 templates<\/code><\/pre>\n\n\n\n
Running AlertManager<\/h3>\n\n\n\n
You can run Alertmanager in standalone mode by executing the command below;<\/p>\n\n\n\n
alertmanager --config.file \/etc\/alertmanager\/alertmanager.yml<\/code><\/pre>\n\n\n\n
Remember, we set Prometheus to connect to Alertmanager via localhost:9093<\/strong><\/code>, hence replace the x.x.x.x<\/strong><\/code> with the correct address.<\/p>\n\n\n\n
To run AlertManager as a service;<\/p>\n\n\n\n
cat > \/etc\/systemd\/system\/alertmanager.service << 'EOL'\n[Unit]\nDescription=AlertManager Server Service\nWants=network-online.target\nAfter=network-online.target\n\n[Service]\nUser=root\nGroup=root\nType=simple\nExecStart=\/usr\/local\/bin\/alertmanager --config.file \/etc\/alertmanager\/alertmanager.yml\n\n[Install]\nWantedBy=multi-user.target\nEOL<\/code><\/pre>\n\n\n\n
For other options, consult \/usr\/local\/bin\/alertmanager --help<\/code><\/strong>.<\/p>\n\n\n\n
Reload systemd configurations and start Alertmanager;<\/p>\n\n\n\n
systemctl daemon-reload<\/code><\/pre>\n\n\n\n
systemctl enable --now alertmanager<\/code><\/pre>\n\n\n\n
Check the status;<\/p>\n\n\n\n
systemctl status alertmanager<\/code><\/pre>\n\n\n\n
\u25cf alertmanager.service - AlertManager Server Service\n   Loaded: loaded (\/etc\/systemd\/system\/alertmanager.service; enabled; vendor preset: enabled)\n   Active: active (running) since Sat 2021-04-10 23:16:25 EAT; 1s ago\n Main PID: 3959 (alertmanager)\n    Tasks: 8 (limit: 2359)\n   Memory: 13.5M\n   CGroup: \/system.slice\/alertmanager.service\n           \u2514\u25003959 \/usr\/local\/bin\/alertmanager --config.file \/etc\/alertmanager\/alertmanager.yml --web.external-url=http:\/\/localhost:9093\n\nApr 10 23:16:25 debian systemd[1]: Started AlertManager Server Service.\n...\n...\nApr 10 23:16:25 debian alertmanager[3959]: level=info ts=2021-04-10T20:16:25.745Z caller=main.go:485 msg=Listening address=:9093<\/code><\/pre>\n\n\n\n
Check Prometheus Alerts page;<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Also check your mail;<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
You can further customize the email template.<\/p>\n\n\n\n
You can also view alerts in Alertmanager, http:\/\/x.x.x.x:9093\/alerts<\/strong>;<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Further Reading<\/h3>\n\n\n\n
AlertManager overview<\/a><\/p>\n\n\n\n
Other Tutorials<\/h3>\n\n\n\n
Monitor SSL\/TLS Certificates Expiry with Nagios<\/a><\/p>\n\n\n\n
Monitor Linux System Metrics with Prometheus Node Exporter<\/a><\/p>\n\n\n\n
Monitoring Gitlab Metrics with Prometheus and Grafana<\/a><\/p>\n\n\n\n
Monitor OpenVPN Connections with Prometheus and Grafana<\/a><\/p>\n\n\n\n
Install OpenNMS Network Monitoring tool on Ubuntu 20.04<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
In this tutorial, you will learn how to configure Prometheus Email alerting with AlertManager. AlertManager is used to handle alerts sent by client applications such<\/p>\n","protected":false},"author":3,"featured_media":8560,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"rank_math_lock_modified_date":false,"footnotes":""},"categories":[72,301,121,294],"tags":[3423,3422,3425,3424],"class_list":["post-8589","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-monitoring","category-grafana","category-howtos","category-prometheus","tag-alertmanager-email-alerting","tag-configure-alerting-with-prometheuss","tag-configure-prometheus-email-alerting-with-alertmanager","tag-prometheus-alertmanager","generate-columns","tablet-grid-50","mobile-grid-100","grid-parent","grid-50","resize-featured-image"],"_links":{"self":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/8589"}],"collection":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/comments?post=8589"}],"version-history":[{"count":5,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/8589\/revisions"}],"predecessor-version":[{"id":21849,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/8589\/revisions\/21849"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media\/8560"}],"wp:attachment":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media?parent=8589"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/categories?post=8589"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/tags?post=8589"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}