{"id":16185,"date":"2023-04-15T09:42:28","date_gmt":"2023-04-15T06:42:28","guid":{"rendered":"https:\/\/kifarunix.com\/?p=16185"},"modified":"2024-03-10T08:55:45","modified_gmt":"2024-03-10T05:55:45","slug":"monitor-docker-swarm-and-container-metrics-using-metricbeat","status":"publish","type":"post","link":"https:\/\/kifarunix.com\/monitor-docker-swarm-and-container-metrics-using-metricbeat\/","title":{"rendered":"Monitor Docker Swarm and Container metrics using Metricbeat"},"content":{"rendered":"\n<p>Is it possible to monitor Docker Swarm and container metrics using Metricbeat? Yes, of course! Metricbeat provides a flexible and powerful way to monitor Docker environment, be it individual containers or Swarm services. It can collect metrics about Docker containers such as container CPU usage, memory usage, network statistics&#8230; and Swarm metrics such as the number of replicas, service health&#8230; The metrics are collected and pushed to Elasticsearch stack for parsing and processing and they can be visualized on Kibana interface.<\/p>\n\n\n\n<div class=\"wp-block-rank-math-toc-block\" id=\"rank-math-toc\"><h2>Table of Contents<\/h2><nav><ul><li><a href=\"#monitoring-docker-swarm-and-container-metrics-using-metricbeat\">Monitoring Docker Swarm and Container Metrics using Metricbeat<\/a><ul><li><a href=\"#install-and-setup-elk-stack\">Install and Setup ELK Stack<\/a><\/li><li><a href=\"#setup-docker-swarm-cluster\">Setup Docker Swarm Cluster<\/a><\/li><li><a href=\"#deploy-metricbeat-on-each-swarm-node\">Deploy Metricbeat on each Swarm Node<\/a><ul><li><a href=\"#install-metricbeat-on-linux-systems\">Install Metricbeat on Linux Systems<\/a><\/li><li><a href=\"#configure-metricbeat\">Configure Metricbeat<\/a><\/li><li><a href=\"#create-metricbeat-keystore-and-add-elasticsearch-connection-username-password\">Create Metricbeat Keystore and Add Elasticsearch Connection Username\/Password<\/a><\/li><\/ul><\/li><li><a href=\"#enable-metricbeat-docker-system-module\">Enable Metricbeat Docker\/System Module<\/a><ul><li><a href=\"#enable-metricbeat-system-module\">Enable Metricbeat System Module<\/a><\/li><li><a href=\"#enable-metricbeat-docker-module\">Enable Metricbeat Docker Module<\/a><\/li><\/ul><\/li><li><a href=\"#test-metricbeat-configuration-modules-and-output\">Test Metricbeat Configuration, Modules and Output<\/a><\/li><li><a href=\"#load-kibana-metricbeat-dashboards\">Load Kibana Metricbeat Dashboards<\/a><\/li><li><a href=\"#running-metricbeat\">Running Metricbeat<\/a><\/li><li><a href=\"#viewing-metricbeat-data\">Viewing Metricbeat Data<\/a><\/li><li><a href=\"#metricbeat-dashboards\">Metricbeat Dashboards<\/a><\/li><li><a href=\"#other-tutorials\">Other Tutorials<\/a><\/li><\/ul><\/li><\/ul><\/nav><\/div>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"monitoring-docker-swarm-and-container-metrics-using-metricbeat\">Monitoring Docker Swarm and Container Metrics using Metricbeat<\/h2>\n\n\n\n<p>In this guide, we will use Metricbeat to monitor Docker swarm and container metrics.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"install-and-setup-elk-stack\">Install and Setup ELK Stack<\/h3>\n\n\n\n<p>You can deploy your ELK stack as Docker container or just run it on the host as usual.<\/p>\n\n\n\n<p>In this tutorial, we are running ELK stack on a different host, not as Docker containers though.<\/p>\n\n\n\n<p>See the links below on how to install and setup ELK;<\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/?s=install+elk\" target=\"_blank\" rel=\"noreferrer noopener\">How to Install ELK Stack on Linux<\/a><\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"setup-docker-swarm-cluster\">Setup Docker Swarm Cluster<\/h3>\n\n\n\n<p>Since we are dealing swarm services, you need to have setup swarm cluster that whose metrics need to be monitored.<\/p>\n\n\n\n<p>If you haven&#8217;t, please refer to this guide to learn how to setup swarm cluster.<\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/how-to-setup-three-node-docker-swarm-cluster-on-ubuntu\/\" target=\"_blank\" rel=\"noreferrer noopener\">How to Setup Docker Swarm Cluster on Ubuntu<\/a><\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"deploy-metricbeat-on-each-swarm-node\">Deploy Metricbeat on each Swarm Node<\/h3>\n\n\n\n<p>In order to collect metrics from each node in the swarm cluster, we will deploy Metricbeat on each node.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"install-metricbeat-on-linux-systems\">Install Metricbeat on Linux Systems<\/h4>\n\n\n\n<p>Since we are running our Docker swarm cluster on Linux systems, let&#8217;s deploy Metricbeat on each node of the swarm cluster.<\/p>\n\n\n\n<p>In order to install Metricbeat on Linux systems, you can follow this guide below;<\/p>\n\n\n\n<p>Note, you need to install a version that is exactly same or a bit lower than the version of your ELK stack. We use version 8.7.0, which is the current new stable release as of this writing, in this guide.<\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/monitor-linux-system-metrics-with-elk-stack\/#install-metricbeat-in-linux-systems\" target=\"_blank\" rel=\"noreferrer noopener\">Install Metricbeat in Linux Systems<\/a><\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"configure-metricbeat\">Configure Metricbeat<\/h4>\n\n\n\n<p>By default, this is how this configuration looks like;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>cat \/etc\/metricbeat\/metricbeat.yml<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>\n###################### Metricbeat Configuration Example #######################\n\n# This file is an example configuration file highlighting only the most common\n# options. The metricbeat.reference.yml file from the same directory contains all the\n# supported options with more comments. You can use it as a reference.\n#\n# You can find the full configuration reference here:\n# https:\/\/www.elastic.co\/guide\/en\/beats\/metricbeat\/index.html\n\n# =========================== Modules configuration ============================\n\nmetricbeat.config.modules:\n  # Glob pattern for configuration loading\n  path: ${path.config}\/modules.d\/*.yml\n\n  # Set to true to enable config reloading\n  reload.enabled: false\n\n  # Period on which files under path should be checked for changes\n  #reload.period: 10s\n\n# ======================= Elasticsearch template setting =======================\n\nsetup.template.settings:\n  index.number_of_shards: 1\n  index.codec: best_compression\n  #_source.enabled: false\n\n\n# ================================== General ===================================\n\n# The name of the shipper that publishes the network data. It can be used to group\n# all the transactions sent by a single shipper in the web interface.\n#name:\n\n# The tags of the shipper are included in their own field with each\n# transaction published.\n#tags: [\"service-X\", \"web-tier\"]\n\n# Optional fields that you can specify to add additional information to the\n# output.\n#fields:\n#  env: staging\n\n# ================================= Dashboards =================================\n# These settings control loading the sample dashboards to the Kibana index. Loading\n# the dashboards is disabled by default and can be enabled either by setting the\n# options here or by using the `setup` command.\n#setup.dashboards.enabled: false\n\n# The URL from where to download the dashboards archive. By default this URL\n# has a value which is computed based on the Beat name and version. For released\n# versions, this URL points to the dashboard archive on the artifacts.elastic.co\n# website.\n#setup.dashboards.url:\n\n# =================================== Kibana ===================================\n\n# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.\n# This requires a Kibana endpoint configuration.\nsetup.kibana:\n\n  # Kibana Host\n  # Scheme and port can be left out and will be set to the default (http and 5601)\n  # In case you specify and additional path, the scheme is required: http:\/\/localhost:5601\/path\n  # IPv6 addresses should always be defined as: https:\/\/[2001:db8::1]:5601\n  #host: \"localhost:5601\"\n\n  # Kibana Space ID\n  # ID of the Kibana Space into which the dashboards should be loaded. By default,\n  # the Default Space will be used.\n  #space.id:\n\n# =============================== Elastic Cloud ================================\n\n# These settings simplify using Metricbeat with the Elastic Cloud (https:\/\/cloud.elastic.co\/).\n\n# The cloud.id setting overwrites the `output.elasticsearch.hosts` and\n# `setup.kibana.host` options.\n# You can find the `cloud.id` in the Elastic Cloud web UI.\n#cloud.id:\n\n# The cloud.auth setting overwrites the `output.elasticsearch.username` and\n# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.\n#cloud.auth:\n\n# ================================== Outputs ===================================\n\n# Configure what output to use when sending the data collected by the beat.\n\n# ---------------------------- Elasticsearch Output ----------------------------\noutput.elasticsearch:\n  # Array of hosts to connect to.\n  hosts: [\"localhost:9200\"]\n\n  # Protocol - either `http` (default) or `https`.\n  #protocol: \"https\"\n\n  # Authentication credentials - either API key or username\/password.\n  #api_key: \"id:api_key\"\n  #username: \"elastic\"\n  #password: \"changeme\"\n\n# ------------------------------ Logstash Output -------------------------------\n#output.logstash:\n  # The Logstash hosts\n  #hosts: [\"localhost:5044\"]\n\n  # Optional SSL. By default is off.\n  # List of root certificates for HTTPS server verifications\n  #ssl.certificate_authorities: [\"\/etc\/pki\/root\/ca.pem\"]\n\n  # Certificate for SSL client authentication\n  #ssl.certificate: \"\/etc\/pki\/client\/cert.pem\"\n\n  # Client Certificate Key\n  #ssl.key: \"\/etc\/pki\/client\/cert.key\"\n\n# ================================= Processors =================================\n\n# Configure processors to enhance or manipulate events generated by the beat.\n\nprocessors:\n  - add_host_metadata: ~\n  - add_cloud_metadata: ~\n  - add_docker_metadata: ~\n  - add_kubernetes_metadata: ~\n\n\n# ================================== Logging ===================================\n\n# Sets log level. The default log level is info.\n# Available log levels are: error, warning, info, debug\n#logging.level: debug\n\n# At debug level, you can selectively enable logging only for some components.\n# To enable all selectors use [\"*\"]. Examples of other selectors are \"beat\",\n# \"publisher\", \"service\".\n#logging.selectors: [\"*\"]\n\n# ============================= X-Pack Monitoring ==============================\n# Metricbeat can export internal metrics to a central Elasticsearch monitoring\n# cluster.  This requires xpack monitoring to be enabled in Elasticsearch.  The\n# reporting is disabled by default.\n\n# Set to true to enable the monitoring reporter.\n#monitoring.enabled: false\n\n# Sets the UUID of the Elasticsearch cluster under which monitoring data for this\n# Metricbeat instance will appear in the Stack Monitoring UI. If output.elasticsearch\n# is enabled, the UUID is derived from the Elasticsearch cluster referenced by output.elasticsearch.\n#monitoring.cluster_uuid:\n\n# Uncomment to send the metrics to Elasticsearch. Most settings from the\n# Elasticsearch output are accepted here as well.\n# Note that the settings should point to your Elasticsearch *monitoring* cluster.\n# Any setting that is not set is automatically inherited from the Elasticsearch\n# output configuration, so if you have the Elasticsearch output configured such\n# that it is pointing to your Elasticsearch monitoring cluster, you can simply\n# uncomment the following line.\n#monitoring.elasticsearch:\n\n# ============================== Instrumentation ===============================\n\n# Instrumentation support for the metricbeat.\n#instrumentation:\n    # Set to true to enable instrumentation of metricbeat.\n    #enabled: false\n\n    # Environment in which metricbeat is running on (eg: staging, production, etc.)\n    #environment: \"\"\n\n    # APM Server hosts to report instrumentation results to.\n    #hosts:\n    #  - http:\/\/localhost:8200\n\n    # API Key for the APM Server(s).\n    # If api_key is set then secret_token will be ignored.\n    #api_key:\n\n    # Secret token for the APM Server(s).\n    #secret_token:\n\n\n# ================================= Migration ==================================\n\n# This allows to enable 6.7 migration aliases\n#migration.6_to_7.enabled: true\n<\/code><\/pre>\n\n\n\n<p>So, we have updated our config to simply look like;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>cat \/etc\/metricbeat\/metricbeat.yml<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>\nmetricbeat.config.modules:\n  path: ${path.config}\/modules.d\/*.yml\n  reload.enabled: false\nsetup.template.settings:\n  index.number_of_shards: 1\n  index.codec: best_compression\noutput.elasticsearch:\n  hosts: [\"node03.kifarunix-demo.com:9200\"]\n  protocol: \"https\"\n  ssl.certificate_authorities: [\"\/etc\/metricbeat\/ca.crt\"]\n  username: \"${ES_USER}\"\n  password: \"${ES_PASSWORD}\"\nprocessors:\n  - add_host_metadata: ~\n  - add_docker_metadata: ~\nlogging.level: info\n<\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>The <code>metricbeat.config.modules<\/code> specifies how Metricbeat should load and reload module configurations.\n<ul class=\"wp-block-list\">\n<li><code>${path.config}\/modules.d\/*.yml<\/code> specifies the path to the directory containing module configuration files<\/li>\n\n\n\n<li> <code>reload.enabled: false<\/code> specifies whether Metricbeat should reload module configurations as they change.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><code>setup.template.settings<\/code>: allows you to customize the index template settings that Metricbeat uses when creating indices in Elasticsearch.\n<ul class=\"wp-block-list\">\n<li>the <code>index.number_of_shards<\/code> setting is set to <code>1<\/code>, which means that each index will have only one primary shard<\/li>\n\n\n\n<li>the <code>index.codec<\/code> setting is set to <code>best_compression<\/code>, which means that Elasticsearch will use the best compression algorithm available for the index.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><code>output.elasticsearch<\/code>: configures the Elasticsearch output for Metricbeat.\n<ul class=\"wp-block-list\">\n<li>The <code>hosts<\/code> setting specifies the Elasticsearch instance to send the data to,<\/li>\n\n\n\n<li>The <code>protocol<\/code> setting specifies the protocol to use (in this case, HTTPS)<\/li>\n\n\n\n<li>The <code>ssl.certificate_authorities<\/code> setting specifies the path to the CA certificate used to verify the Elasticsearch server&#8217;s SSL\/TLS certificate.<\/li>\n\n\n\n<li>The <code>username<\/code> and <code>password<\/code> settings specify the credentials to use for authentication. The values of the variables used are stored in the Metricbeat keystore, to be created later below.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><code>setup.ilm.enable: true<\/code> Enables or disables index lifecycle management on any new indices created by Metricbeat.<\/li>\n\n\n\n<li><code>setup.ilm.rollover_alias: \"metricbeat\"<\/code>: This specifies the alias that is used for the ILM rollover process.<\/li>\n\n\n\n<li><code>setup.ilm.pattern: \"{now\/d}-000001\"<\/code>: This sets the name pattern for the index to something like <strong><code>metricbeat-2023.04.14-000001<\/code><\/strong>.<\/li>\n\n\n\n<li><code>processors<\/code>: This section defines the processors to apply to each event before sending it to Elasticsearch. In this example;\n<ul class=\"wp-block-list\">\n<li>the <code>add_host_metadata<\/code> processor adds metadata about the host where the event was generated,<\/li>\n\n\n\n<li>the <code>add_docker_metadata<\/code> processor adds metadata about the Docker container running on the host, if applicable.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><code>logging.level<\/code>: This setting controls the logging level for Metricbeat.\n<ul class=\"wp-block-list\">\n<li>the logging level is set to <code>info<\/code>, which means that Metricbeat will log information and above (i.e., warnings and errors).<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"create-metricbeat-keystore-and-add-elasticsearch-connection-username-password\">Create Metricbeat Keystore and Add Elasticsearch Connection Username\/Password<\/h4>\n\n\n\n<p>If you want, you can leave the username and password in plain-text. Otherwise, create a keystore and store the username and password using variables as defined in our config above.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>metricbeat keystore create<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>metricbeat keystore add ES_USER<\/code><\/pre>\n\n\n\n<p>Enter the username that has rights to create and write to index to be used with Metricbeat.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>metricbeat keystore add ES_PASSWORD<\/code><\/pre>\n\n\n\n<p>Enter the password for the above user.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"enable-metricbeat-docker-system-module\">Enable Metricbeat Docker\/System Module<\/h3>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"enable-metricbeat-system-module\">Enable Metricbeat System Module<\/h4>\n\n\n\n<p>System module, that enables you to monitor host server metrics such as <code>cpu<\/code>,&nbsp;<code>load<\/code>,&nbsp;<code>memory<\/code>,&nbsp;<code>network<\/code>,&nbsp;<code>process<\/code>,&nbsp;<code>process_summary<\/code>,&nbsp;<code>socket_summary<\/code>,&nbsp;<code>filesystem<\/code>,&nbsp;<code>fsstat<\/code>, and&nbsp;<code>uptime<\/code> is enabled by default.<\/p>\n\n\n\n<p>Confirm by checking that under the modules directory, the <strong><code>system.yml<\/code><\/strong> has no <strong><code>.disabled<\/code><\/strong> suffix. You can also list the modules.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>metricbeat modules list<\/code><\/pre>\n\n\n\n<p>We will configure the default <a href=\"https:\/\/www.elastic.co\/guide\/en\/beats\/metricbeat\/current\/metricbeat-module-system.html\" target=\"_blank\" rel=\"noreferrer noopener\">system module<\/a> as follows;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>vim \/etc\/metricbeat\/modules.d\/system.yml<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>\n- module: system\n  period: 10s\n  metricsets:\n    - cpu\n    - load\n    - memory\n    - network\n    - process\n    - process_summary\n    - socket_summary\n    - service\n    - users\n  process.include_top_n:\n    by_cpu: 5      \n    by_memory: 5\n- module: system\n  period: 1m\n  metricsets:\n    - filesystem  \n    - fsstat\n  processors:\n  - drop_event.when.regexp:\n      system.filesystem.mount_point: '^\/(sys|cgroup|proc|dev|etc|host|lib|snap)($|\/)'\n- module: system\n  period: 15m\n  metricsets:\n    - uptime\n<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"enable-metricbeat-docker-module\">Enable Metricbeat Docker Module<\/h4>\n\n\n\n<p>Enable and configure <a href=\"https:\/\/www.elastic.co\/guide\/en\/beats\/metricbeat\/current\/metricbeat-module-docker.html\" target=\"_blank\" rel=\"noreferrer noopener\">Docker module<\/a>;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>metricbeat modules enable docker<\/code><\/pre>\n\n\n\n<p>Here is our configuration;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>vim \/etc\/metricbeat\/modules.d\/docker.yml<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>\n- module: docker\n  metricsets:\n    - container\n    - cpu\n    - diskio\n    - event\n    - healthcheck\n    - info\n    - memory\n    - network\n    - network_summary\n  period: 5s\n  hosts: [\"unix:\/\/\/var\/run\/docker.sock\"]\n\n  labels.dedot: false\n  processors:\n    - add_docker_metadata: ~\n<\/code><\/pre>\n\n\n\n<p><strong>hosts<\/strong> parameter enables the module to connect to Docker daemon socket to collect Docker related metrics defined on the metricsset above.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"test-metricbeat-configuration-modules-and-output\">Test Metricbeat Configuration, Modules and Output<\/h3>\n\n\n\n<p>Run the commands below to test Metricbeat configuration, module settings and to check if Metricbeat can connect to the output defined with the current settings, respectively;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>metricbeat test config<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>metricbeat test modules<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>metricbeat test output<\/code><\/pre>\n\n\n\n<p>If there is any issue, please fix it before you can proceed.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"load-kibana-metricbeat-dashboards\">Load Kibana Metricbeat Dashboards<\/h3>\n\n\n\n<p>Each and every module comes with predefined dashboards, visualizations, and searches for visualizing Metricbeat data in Kibana.<\/p>\n\n\n\n<p>To utilize these dashboards on Kibana, you need to load them onto Kibana, as shown below. <\/p>\n\n\n\n<p>Before you can proceed, ensure that Kibana is running and reachable.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>nc -vz node01.kifarunix-demo.com 5601<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code><strong>Connection to node01.kifarunix-demo.com (192.168.58.22) 5601 port &#91;tcp\/*] succeeded!<\/strong><\/code><\/pre>\n\n\n\n<p>Note, we are using Elasticsearch as output hence;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>metricbeat setup --dashboards \\\n-E 'setup.kibana.host=node01.kifarunix-demo.com:5601' \\\n-E 'setup.dashboards.enabled=true'<\/code><\/pre>\n\n\n\n<p>If you have configured Metricbeat to write to a custom index pattern apart from the default, metricbeat-*, the you can specify custom one using the <strong><code>setup.dashboards.index<\/code><\/strong>. This setting overwrites the index name defined in the dashboards and index pattern.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>metricbeat setup --dashboards \\\n-E 'setup.kibana.host=node01.kifarunix-demo.com:5601' \\\n-E 'setup.dashboards.enabled=true' \\\n-E 'setup.dashboards.index=<strong>custom-*<\/strong>'<\/code><\/pre>\n\n\n\n<p>The <strong><code>metricbeat setup --dashboards<\/code><\/strong> command does initial setup of the environment:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Index mapping template in Elasticsearch to ensure fields are mapped.<\/li>\n\n\n\n<li>Kibana dashboards (where available).<\/li>\n\n\n\n<li>Ingest pipelines (where available).<\/li>\n\n\n\n<li>ILM policy (for Elasticsearch 6.5 and newer).<\/li>\n<\/ul>\n\n\n\n<p>When you see, <strong><code>Loaded dashboards<\/code><\/strong>, you are good to go.<\/p>\n\n\n\n<p>You can confirm from Kibana if there Kibana index patterns\/data view (Management\/Kibana\/Data Views) for Metricbeat is created already.<\/p>\n\n\n\n<p>Check dashboards from the Dashboards.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"running-metricbeat\">Running Metricbeat<\/h3>\n\n\n\n<p>To begin with, let&#8217;s run Metricbeat in foreground;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>metricbeat -e<\/code><\/pre>\n\n\n\n<p>If all is good, you should see a line like;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Connection to backoff(elasticsearch(ES_ADDRESS:9200)) established<\/code><\/pre>\n\n\n\n<p>You can cancel foreground and start and enable Metricbeat to run on system boot;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>systemctl enable --now metricbeat<\/code><\/pre>\n\n\n\n<p>On the rest of the nodes on the cluster, install and configure Metricbeat as done above. You dont need to load dashboards again, once you have loaded on one node.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"viewing-metricbeat-data\">Viewing Metricbeat Data<\/h3>\n\n\n\n<p>Metricbeat Index is hidden, click show hidden indices to see it;<\/p>\n\n\n\n<div><a href=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/04\/metricbeat-hidden-index.png\" class=\"td-modal-image\"><figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1884\" height=\"646\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/04\/metricbeat-hidden-index.png\" alt=\"Monitor Docker Swarm and Container metrics using Metricbeat\" class=\"wp-image-16197\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/04\/metricbeat-hidden-index.png?v=1681506658 1884w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/04\/metricbeat-hidden-index-768x263.png?v=1681506658 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/04\/metricbeat-hidden-index-1536x527.png?v=1681506658 1536w\" sizes=\"(max-width: 1884px) 100vw, 1884px\" \/><\/figure><\/a><\/div>\n\n\n\n<p>The Kibana data view is automatically created when you load dashboards.<\/p>\n\n\n\n<p>To see raw metrics data, navigate to Kibana Discover menu and select respective pattern;<\/p>\n\n\n\n<div><a href=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/04\/metricbeat-raw-metrics.png\" class=\"td-modal-image\"><figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1907\" height=\"934\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/04\/metricbeat-raw-metrics.png\" alt=\"Monitor Docker Swarm and Container metrics using Metricbeat\" class=\"wp-image-16199\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/04\/metricbeat-raw-metrics.png?v=1681507685 1907w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/04\/metricbeat-raw-metrics-768x376.png?v=1681507685 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/04\/metricbeat-raw-metrics-1536x752.png?v=1681507685 1536w\" sizes=\"(max-width: 1907px) 100vw, 1907px\" \/><\/figure><\/a><\/div>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"metricbeat-dashboards\">Metricbeat Dashboards<\/h3>\n\n\n\n<p>Metric System Dashboards sample;<\/p>\n\n\n\n<div><a href=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/04\/system-host-overview-metrics.png\" class=\"td-modal-image\"><figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1890\" height=\"3511\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/04\/system-host-overview-metrics.png\" alt=\"Monitor Docker Swarm and Container metrics using Metricbeat\" class=\"wp-image-16204\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/04\/system-host-overview-metrics.png?v=1681540642 1890w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/04\/system-host-overview-metrics-768x1427.png?v=1681540642 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/04\/system-host-overview-metrics-827x1536.png?v=1681540642 827w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/04\/system-host-overview-metrics-1102x2048.png?v=1681540642 1102w\" sizes=\"(max-width: 1890px) 100vw, 1890px\" \/><\/figure><\/a><\/div>\n\n\n\n<p>Docker overview metrics. You can filter by the agent name, e.g <code><strong>agent.name: \"node-name\"<\/strong><\/code> to display information about Docker containers running on each swarm node.<\/p>\n\n\n\n<div><a href=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/04\/docker-overview-metrics.png\" class=\"td-modal-image\"><figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1900\" height=\"1730\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/04\/docker-overview-metrics.png\" alt=\"Monitor Docker Swarm and Container metrics using Metricbeat\" class=\"wp-image-16203\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/04\/docker-overview-metrics.png?v=1681540603 1900w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/04\/docker-overview-metrics-768x699.png?v=1681540603 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/04\/docker-overview-metrics-1536x1399.png?v=1681540603 1536w\" sizes=\"(max-width: 1900px) 100vw, 1900px\" \/><\/figure><\/a><\/div>\n\n\n\n<p>And that is it. You can create your own custom dashboards to suite you requirements.<\/p>\n\n\n\n<p>That marks the end of our tutorial on how to monitoring Docker Swarm and container metrics using Metricbeat.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"other-tutorials\">Other Tutorials<\/h3>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/monitor-docker-containers-metrics-using-grafana\/\" target=\"_blank\" rel=\"noreferrer noopener\">Monitor Docker Containers Metrics using Grafana<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/connect-to-remote-docker-environment-on-docker-desktop\/\" target=\"_blank\" rel=\"noreferrer noopener\">Connect to Remote Docker Environment on Docker Desktop<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Is it possible to monitor Docker Swarm and container metrics using Metricbeat? Yes, of course! Metricbeat provides a flexible and powerful way to monitor Docker<\/p>\n","protected":false},"author":10,"featured_media":16203,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"rank_math_lock_modified_date":false,"footnotes":""},"categories":[72,1076,1077,910,121],"tags":[6531,6528,6529,6527,6530],"class_list":["post-16185","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-monitoring","category-containers","category-docker","category-elastic-stack","category-howtos","tag-docker-metrics-monitoring","tag-docker-swarm-metrics-with-metricbeat","tag-install-metricbeat","tag-monitor-docker-container-metrics-using-metricbeat","tag-monitor-docker-with-elk","generate-columns","tablet-grid-50","mobile-grid-100","grid-parent","grid-50","resize-featured-image"],"_links":{"self":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/16185"}],"collection":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/users\/10"}],"replies":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/comments?post=16185"}],"version-history":[{"count":18,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/16185\/revisions"}],"predecessor-version":[{"id":20726,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/16185\/revisions\/20726"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media\/16203"}],"wp:attachment":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media?parent=16185"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/categories?post=16185"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/tags?post=16185"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}