{"id":16042,"date":"2023-04-07T14:13:16","date_gmt":"2023-04-07T11:13:16","guid":{"rendered":"https:\/\/kifarunix.com\/?p=16042"},"modified":"2024-03-10T09:02:05","modified_gmt":"2024-03-10T06:02:05","slug":"monitor-docker-swarm-node-metrics-using-grafana","status":"publish","type":"post","link":"https:\/\/kifarunix.com\/monitor-docker-swarm-node-metrics-using-grafana\/","title":{"rendered":"Monitor Docker Swarm Node Metrics using Grafana"},"content":{"rendered":"\n

How can I monitor Docker swarm node metrics? In this tutorial, you will learn how to monitor Docker swarm node metrics using Grafana. Prometheus node exporter can be used to collect hardware and OS metrics that are exposed by the kernel. Such metrics are scraped using Promethes, which acts as data source to Grafana which let’s you create visualization for all your Docker swarm node metrics for monitoring.<\/p>\n\n\n\n

Monitoring Docker Swarm Node Metrics using Grafana<\/h2>\n\n\n\n

In this guide, we will use Grafana, Prometheus and Node Exporter to monitor Docker swarm node metrics. <\/p>\n\n\n\n

We will run Grafana and Prometheus as swarm services, while Node exporter as separate docker containers on each swarm node.<\/p>\n\n\n\n

Install Docker Engine<\/h3>\n\n\n\n

If you have not already installed Docker Engine, please refer to appropriate guides in the link below;<\/p>\n\n\n\n

Install Docker in Linux<\/a><\/p>\n\n\n\n

Setup Docker Swarm Cluster<\/h3>\n\n\n\n

Since we are dealing swarm services, you need to have setup swarm cluster that whose metrics need to be monitored.<\/p>\n\n\n\n

If you haven’t, please refer to this guide to learn how to setup swarm cluster.<\/p>\n\n\n\n

How to Setup Docker Swarm Cluster on Ubuntu<\/a><\/p>\n\n\n\n

Create Docker Swarm Network to Interconnect Monitoring Tools <\/h3>\n\n\n\n

The monitoring tools should be in the same network for them to communicate with each other.<\/p>\n\n\n\n

Since we are dealing with Docker swarm, then overlay networks can be used to provide communication between services.<\/p>\n\n\n\n

If you need to configure Docker containers to use Docker swarm network, then you need to make the Docker swarm network you are creating to be attachable<\/strong>.<\/p>\n\n\n\n

We have already created an overlay network called monitoring_stack<\/strong><\/code>.<\/p>\n\n\n\n

You check available Docker networks using the command below;<\/p>\n\n\n\n

docker network ls<\/code><\/pre>\n\n\n\n
\nNETWORK ID     NAME               DRIVER    SCOPE\n816c7f48bbb3   bridge             bridge    local\n841640b7368f   docker_gwbridge    bridge    local\n45c3672051a1   host               host      local\nsxnbs83rwn5f   ingress            overlay   swarm\ne5mklugaqapm   monitoring_stack   overlay   swarm<\/strong>\n05990914584e   none               null      local\n<\/code><\/pre>\n\n\n\n
You can use this command to create the network, replacing monitoring_stack<\/strong> with your preferred network name.<\/p>\n\n\n\n
docker network create --driver overlay --attachable monitoring_stack<\/code><\/pre>\n\n\n\n
Deploy Node Exporter Docker Container<\/h3>\n\n\n\n
On each Docker Swarm cluster node, deploy the Prometheus Node Exporter to collect individual node metrics;<\/p>\n\n\n\n
We have three nodes in our swarm cluster;<\/p>\n\n\n\n
docker node ls<\/code><\/pre>\n\n\n\n
\nID                            HOSTNAME   STATUS    AVAILABILITY   MANAGER STATUS   ENGINE VERSION\nhim49eblt74amba1nghv2f8z1 *   swarm01    Ready     Active         Leader           20.10.22\n4lxyd0f6d9h039y5itjiffqax     swarm02    Ready     Active         Reachable        20.10.12\ndfmdl4wy4e7ouklu8mu7nqqh9     swarm03    Ready     Active         Reachable        20.10.22\n<\/code><\/pre>\n\n\n\n
You can simply run the command below to deploy Node Exporter and attach it to our custom Docker network above;<\/p>\n\n\n\n
To deploy Node Exporter Docker container on our Swarm Node01;<\/p>\n\n\n\n
\ndocker run -d \\\n  --name node-swarm01 \\\n  --restart unless-stopped \\\n  --network monitoring_stack \\\n  --volume \/:\/rootfs:ro \\\n  --volume \/var\/run:\/var\/run:rw \\\n  --volume \/sys:\/sys:ro \\\n  --volume \/var\/lib\/docker\/:\/var\/lib\/docker:ro \\\n  gcr.io\/cadvisor\/cadvisor:v0.47.1\n<\/code><\/pre>\n\n\n\n
On the rest of the nodes, run the command, replacing the name of the node exporter on each node.<\/p>\n\n\n\n
By default, Node Exporter exposes metrics on port 9100. This port will be reachable within container networks.<\/p>\n\n\n\n
You can check the status of the Node exporter container by running the command below on each node;<\/p>\n\n\n\n
docker ps<\/code><\/pre>\n\n\n\n
Also possible to get logs;<\/p>\n\n\n\n
docker logs -f node-exp-swarm01<\/code><\/pre>\n\n\n\n
Deploy Prometheus Docker Swarm Service<\/h3>\n\n\n\n
Next, let’s deploy Prometheus as a Docker swarm service to scrape collected swarm node metrics from Node Exporter containers.<\/p>\n\n\n\n
First of all, before you can deploy Prometheus swarm service, create a Prometheus configuration file.<\/p>\n\n\n\n
In our setup, we placed the Prometheus configuration file under \/opt\/prometheus<\/strong><\/code> directory. While creating Prometheus swarm service, we will mount this configuration file to the default Prometheus configuration file, \/etc\/prometheus\/prometheus.yml<\/code><\/strong>.<\/p>\n\n\n\n
vim \/opt\/prometheus\/prometheus.yml<\/code><\/pre>\n\n\n\n
\nglobal:\n  scrape_interval: 5s\n  evaluation_interval: 10s\n\nscrape_configs:\n  - job_name: 'cadvisor'\n    metrics_path: '\/metrics'\n    static_configs:\n      - targets: ['cadvisor-swarm01:8080','cadvisor-swarm02:8080','cadvisor-swarm03:8080']\n  - job_name: 'node-exporter'\n    metrics_path: '\/metrics'\n    static_configs:\n      - targets: ['node-exp-swarm01:9100','node-exp-swarm02:9100','node-exp-swarm03:9100']<\/strong>\n<\/code><\/pre>\n\n\n\n
Prometheus will publish the metrics it scrapes from Node exporter and cAdvisor on port 9090.<\/p>\n\n\n\n
Thus, let’s create Prometheus Docker container using the official Prometheus Docker image and configure it to use our custom network created above, monitoring_stack<\/code>.<\/p>\n\n\n\n
\ndocker service create \\\n-p 9090:9090 \\\n--mode global \\\n--network monitoring_stack \\\n--mount type=bind,src=\/opt\/prometheus\/prometheus.yml,dst=\/etc\/prometheus\/prometheus.yml \\\n--name prometheus \\\nprom\/prometheus\n<\/code><\/pre>\n\n\n\n
If Prometheus is already running as a swarm service then update it;<\/p>\n\n\n\n
docker service update --force prometheus<\/code><\/pre>\n\n\n\n
Your Prometheus container should now be running and exposed via port 9090\/tcp.<\/p>\n\n\n\n
docker service ls --format '{{.Name}}\\t{{.Ports}}'<\/code><\/pre>\n\n\n\n
\nnagios-server\t*:8081->80\/tcp\nprometheus\t*:9090->9090\/tcp<\/strong>\nwordpress\t*:8880->80\/tcp\n<\/code><\/pre>\n\n\n\n
Similarly, ensure you allow port 9090\/tcp on firewall to allow you access the metrics externally!<\/p>\n\n\n\n
Verify Prometheus Targets\/Metrics from Prometheus Dashboard<\/h3>\n\n\n\n
You can now navigate to http:\/\/docker-host-IP:9090\/targets<\/code><\/strong> to see Prometheus targets,<\/p>\n\n\n\n
docker-host-IP<\/code><\/strong> can be IP of any of the swarm node.<\/p>\n\n\n\n
Targets;<\/p>\n\n\n\n
\"Monitor<\/figure><\/a><\/div>\n\n\n\n
Check Swarm Nodes Metrics on Prometheus<\/h3>\n\n\n\n
On the dashboard, click Prometheus,<\/strong> and open metrics explorer and just type container<\/strong> and you should see quite a number of metrics;<\/p>\n\n\n\n
\"Monitor<\/figure><\/a><\/div>\n\n\n\n
For example, you can get Swarm nodes uptime;<\/p>\n\n\n\n
(node_time_seconds - node_boot_time_seconds)\/3600<\/code><\/pre>\n\n\n\n
\"\"<\/figure><\/a><\/div>\n\n\n\n
Deploy Grafana Docker Swarm Service<\/h3>\n\n\n\n
You need Grafana Docker swarm service deployed on your cluster to visualize the swarm cluster metrics.<\/p>\n\n\n\n
Create Grafana Data, Configuration and home directories volumes. Using a volume for Grafana data ensures that the dashboards, users, and other settings you create are persisted across container restarts. This is important if you want to keep your dashboards and other Grafana settings even if the container is updated or recreated.<\/p>\n\n\n\n
for i in conf data home; do docker volume create grafana-$i; done<\/code><\/pre>\n\n\n\n
To deploy Grafana Docker container using the official image, grafana\/grafana-oss<\/code><\/strong> and attach it to the custom network created above.<\/p>\n\n\n\n
\ndocker service create \\\n  --name grafana \\\n  --network monitoring_stack \\\n  --mount type=volume,source=grafana-data,target=\/var\/lib\/grafana \\\n  --mount type=volume,source=grafana-home,target=\/usr\/share\/grafana \\\n  --mount type=volume,source=grafana-config,target=\/etc\/grafana \\\n  --publish published=3000,target=3000 \\\n  grafana\/grafana\n<\/code><\/pre>\n\n\n\n
Grafana is now running as a single replica on all nodes.<\/p>\n\n\n\n
If you run it as global or multi-replicated, you may not be able to login, getting Unauthorized, (user token not found issue in the logs!)<\/p>\n\n\n\n
Open port 3000\/tcp on firewall to allow external access to Grafana;<\/p>\n\n\n\n
ufw allow 3000\/tcp<\/code><\/pre>\n\n\n\n
Accessing Grafana Web Interface<\/h3>\n\n\n\n
You can now access Grafana web interface http:\/\/docker-host-IP:3000.<\/p>\n\n\n\n
Default credentials are admin\/admin<\/strong>. Reset the password and proceed to the Dashboard;<\/p>\n\n\n\n
\"Monitor<\/figure><\/a><\/div>\n\n\n\n
Integrate Prometheus with Grafana For Monitoring<\/h3>\n\n\n\n
You can now integrate Prometheus with Grafana by adding Prometheus data source to Grafana. Check the link below;<\/p>\n\n\n\n
Integrate Prometheus with Grafana For Monitoring<\/a><\/p>\n\n\n\n
Data sources, once you have added the Prometheus data source.<\/p>\n\n\n\n
As you can see, we specified the name of the Prometheus container on the data source URL because both Grafana and Prometheus are on same network.<\/p>\n\n\n\n
\"Monitor<\/figure><\/a><\/div>\n\n\n\n
Create Docker Swarm Node Metrics Dashboards on Grafana<\/h3>\n\n\n\n
You can now create your own Grafana visualization dashboards for your Docker swarm cluster.<\/p>\n\n\n\n
As a simple example, let’s create a simple visualization to check swarm nodes uptime<\/p>\n\n\n\n
on the nodes, you would usually display system uptime using uptime<\/code><\/strong> command or w<\/code> command.<\/p>\n\n\n\n
e.g<\/p>\n\n\n\n
uptime<\/code><\/pre>\n\n\n\n
To create a Grafana dashboard, navigate to dashboards menu > New dashboard.<\/p>\n\n\n\n
\"\"<\/figure><\/a><\/div>\n\n\n\n
Add new panel, we are using Stats<\/strong> panel in this example.<\/p>\n\n\n\n
You have three sections on the dashboard panel;<\/p>\n\n\n\n
\"\"<\/figure><\/a><\/div>\n\n\n\n
So, to create a visualization of specific metric, you need a query to fetch those metrics from the datasource, which in this case is our Preom<\/p>\n\n\n\n