{"id":16802,"date":"2023-05-24T12:40:10","date_gmt":"2023-05-24T09:40:10","guid":{"rendered":"https:\/\/kifarunix.com\/?p=16802"},"modified":"2024-03-10T09:07:36","modified_gmt":"2024-03-10T06:07:36","slug":"kubernetes-monitoring-with-prometheus-and-grafana","status":"publish","type":"post","link":"https:\/\/kifarunix.com\/kubernetes-monitoring-with-prometheus-and-grafana\/","title":{"rendered":"Step-by-Step Guide: Kubernetes Monitoring with Prometheus and Grafana"},"content":{"rendered":"\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1893\" height=\"941\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/05\/kubernetes-cluster-dashboard-metrics.png\" alt=\"\" class=\"wp-image-16827\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/05\/kubernetes-cluster-dashboard-metrics.png?v=1684920721 1893w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/05\/kubernetes-cluster-dashboard-metrics-768x382.png?v=1684920721 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/05\/kubernetes-cluster-dashboard-metrics-1536x764.png?v=1684920721 1536w\" sizes=\"(max-width: 1893px) 100vw, 1893px\" \/><\/figure>\n\n\n\n<p>In this tutorial, we will provide you with a step by step guide on Kubernetes monitoring with Prometheus and Grafana. Being able to monitor a Kubernetes cluster is fundamental in ensuring the health of the cluster, performance and scalability. You can use <a href=\"https:\/\/prometheus.io\/\" target=\"_blank\" rel=\"noreferrer noopener\">Prometheus<\/a> and <a href=\"https:\/\/grafana.com\/\" target=\"_blank\" rel=\"noreferrer noopener\">Grafana<\/a> to provide real-time visibility into your cluster&#8217;s metrics usage. With real time monitoring, you can be able to identify bottlenecks, and optimize resource utilization in the cluster.<\/p>\n\n\n\n<div class=\"wp-block-rank-math-toc-block\" id=\"rank-math-toc\"><h2>Table of Contents<\/h2><nav><ul><li><a href=\"#step-by-step-guide-monitoring-kubernetes-with-prometheus-and-grafana\">Step-by-Step Guide: Monitoring Kubernetes with Prometheus and Grafana<\/a><ul><li><a href=\"#setup-kubernetes-cluster\">Setup Kubernetes Cluster<\/a><\/li><li><a href=\"#install-helm-on-kubernetes-cluster\">Install Helm on Kubernetes Cluster<\/a><\/li><li><a href=\"#install-prometheus-and-grafana-helm-charts-repositories\">Install Prometheus and Grafana Helm Charts Repositories<\/a><\/li><li><a href=\"#install-prometheus-and-grafana-on-kubernetes-cluster\">Install Prometheus and Grafana on Kubernetes Cluster<\/a><\/li><li><a href=\"#accessing-prometheus-outside-k-8-s-cluster\">Accessing Prometheus Outside K8S cluster<\/a><\/li><li><a href=\"#accessing-grafana-outside-k-8-s-cluster\">Accessing Grafana Outside K8S Cluster<\/a><\/li><li><a href=\"#login-to-grafana-web-interface\">Login to Grafana Web Interface<\/a><\/li><li><a href=\"#grafana-prometheus-datasource\">Grafana Prometheus Datasource<\/a><\/li><li><a href=\"#grafana-kubernetes-dashboards\">Grafana Kubernetes Dashboards<\/a><\/li><\/ul><\/li><li><a href=\"#other-tutorials\">Other Tutorials<\/a><\/li><\/ul><\/nav><\/div>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"step-by-step-guide-monitoring-kubernetes-with-prometheus-and-grafana\">Step-by-Step Guide: Monitoring Kubernetes with Prometheus and Grafana<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"setup-kubernetes-cluster\">Setup Kubernetes Cluster<\/h3>\n\n\n\n<p>Of course, you cannot start to monitor what is not setup already. However, if you are looking at how to setup Kubernetes cluster, then check the guide below;<\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/setup-kubernetes-cluster-on-ubuntu\/\" target=\"_blank\" rel=\"noreferrer noopener\">Setup Kubernetes Cluster on Ubuntu 22.04\/20.04<\/a><\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"install-helm-on-kubernetes-cluster\">Install Helm on Kubernetes Cluster<\/h3>\n\n\n\n<p>There are different methods in which you can use to install Kubernetes cluster monitoring tools;<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Creating seperate YAML files for Kubernetes application resources such as deployments, services, pods, etc.<\/li>\n\n\n\n<li>Using Helm Charts. Helm charts are packages of Kubernetes resources that have been created by Helm community to make installation of various K8s packages easy and convenient.<\/li>\n\n\n\n<li>Using Kubernetes operators to automate application deployment.<\/li>\n\n\n\n<li>e.t.c<\/li>\n<\/ol>\n\n\n\n<p>In this tutorial, we will be using Helm charts to deploy Prometheus and Grafana. In that case, you need to have Helm client installed.<\/p>\n\n\n\n<p>Follow the guide below to learn how to install Helm on Kubernetes cluster.<\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/install-helm-on-kubernetes-cluster\/\" target=\"_blank\" rel=\"noreferrer noopener\">How to Install Helm on Kubernetes Cluster<\/a><\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"install-prometheus-and-grafana-helm-charts-repositories\">Install Prometheus and Grafana Helm Charts Repositories<\/h3>\n\n\n\n<p>in order to be able to install Prometheus and Grafana charts on Kubernetes cluster, you first need to install their Helm charts community repositories.<\/p>\n\n\n\n<p>Install Prometheus Helm charts repositories;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>helm repo add prometheus-community https:\/\/prometheus-community.github.io\/helm-charts<\/code><\/pre>\n\n\n\n<p>Install Grafana Helm charts repositories;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>helm repo add grafana https:\/\/grafana.github.io\/helm-charts<\/code><\/pre>\n\n\n\n<p>Confirm that the repos are in place;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>helm repo list<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>NAME                \tURL                                               \nprometheus-community\thttps:\/\/prometheus-community.github.io\/helm-charts\ngrafana             \thttps:\/\/grafana.github.io\/helm-charts             \nbitnami             \thttps:\/\/charts.bitnami.com\/bitnami <\/code><\/pre>\n\n\n\n<p>You can search for any Kubernetes helm chart on the <a href=\"https:\/\/artifacthub.io\/\" target=\"_blank\" rel=\"noreferrer noopener\">K8s charts hub<\/a>.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"install-prometheus-and-grafana-on-kubernetes-cluster\">Install Prometheus and Grafana on Kubernetes Cluster<\/h3>\n\n\n\n<p>Prometheus can collect and store metrics from a variety of sources while Grafana helps you visualize the metrics collected by Prometheus.<\/p>\n\n\n\n<p>There are different charts related to Prometheus\/Grafana on the Artifact Hub that offers different functionality.<\/p>\n\n\n\n<p>In this guide, we will install <a href=\"https:\/\/artifacthub.io\/packages\/helm\/prometheus-community\/kube-prometheus-stack\" target=\"_blank\" rel=\"noreferrer noopener\">kube-prometheus-stack<\/a> chart which offers a complete monitoring solution for K8s cluster. kube-prometheus-stack installs the following components;<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong><code>Prometheus Operator<\/code><\/strong>: The Prometheus Operator is a Kubernetes-native operator that manages and automates the lifecycle of Prometheus and related monitoring components. It simplifies the deployment, configuration, and management of Prometheus instances in a Kubernetes environment.<\/li>\n\n\n\n<li>Highly available <strong><code>Prometheus<\/code><\/strong> which scrapes metrics from various endpoints in the cluster.<\/li>\n\n\n\n<li>Highly available <strong><code>Alertmanager<\/code><\/strong>: The Alertmanager is responsible for processing and managing alerts generated by Prometheus. It allows you to define alerting rules and configure how alerts are sent and handled. The highly available Alertmanager component ensures the reliability and availability of the alerting system.<\/li>\n\n\n\n<li><strong><code>Prometheus node-exporter<\/code><\/strong>: The Prometheus node-exporter is an agent that runs on each Kubernetes node and exposes system-level metrics, such as CPU usage, memory usage, disk utilization, and network statistics. which can be scraped by Prometheus<\/li>\n\n\n\n<li>The <strong><code>Prometheus Adapter<\/code><\/strong> which allows you to use custom and external metrics collected by Prometheus in Kubernetes Horizontal Pod Autoscaling (HPA) and other scaling mechanisms. It enables the Kubernetes API server to retrieve metric values from Prometheus and make scaling decisions based on those metrics.<\/li>\n\n\n\n<li><strong><code>kube-state-metrics<\/code><\/strong> a component which exposes metrics about the state of Kubernetes objects, such as pods, deployments, services, and nodes, providing insights into the current state and health of your Kubernetes resources.<\/li>\n\n\n\n<li><strong><code>Grafana<\/code><\/strong>, the visualization tool itself.<\/li>\n<\/ol>\n\n\n\n<p>To install kube-prometheus-stack chart, run the command below<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>helm install prometheus prometheus-community\/kube-prometheus-stack<\/code><\/pre>\n\n\n\n<p>Once the installation is complete, you will see such an output;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>\nNAME: prometheus\nLAST DEPLOYED: Tue May 23 19:30:09 2023\nNAMESPACE: default\nSTATUS: deployed\nREVISION: 1\nNOTES:\nkube-prometheus-stack has been installed. Check its status by running:\n  kubectl --namespace default get pods -l \"release=prometheus\"\n\nVisit https:\/\/github.com\/prometheus-operator\/kube-prometheus for instructions on how to create & configure Alertmanager and Prometheus instances using the Operator.\n<\/code><\/pre>\n\n\n\n<p>So, what resources for Prometheus stack are installed? You can get using the <strong><code>kubectl get<\/code><\/strong> command as follows;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl get all --selector release=prometheus<\/code><\/pre>\n\n\n\n<p>The command will display the <strong><code>pods<\/code><\/strong>, <strong><code>services<\/code><\/strong>, <strong><code>daemonsets<\/code><\/strong>, <strong><code>deployments<\/code><\/strong>, <strong><code>replicatsets<\/code><\/strong>, <strong><code>statefulsets<\/code><\/strong> related to Prometheus.<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>\nNAME                                                       READY   STATUS    RESTARTS   AGE\npod\/prometheus-kube-prometheus-operator-54cfc96db7-r6k6k   1\/1     Running   0          3m34s\npod\/prometheus-kube-state-metrics-5f5f8b8fdd-nzwsp         1\/1     Running   0          3m34s\npod\/prometheus-prometheus-node-exporter-92wtx              1\/1     Running   0          3m34s\npod\/prometheus-prometheus-node-exporter-c2zdq              1\/1     Running   0          3m34s\npod\/prometheus-prometheus-node-exporter-f6867              1\/1     Running   0          3m34s\npod\/prometheus-prometheus-node-exporter-wf8qd              1\/1     Running   0          3m34s\n\nNAME                                              TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE\nservice\/prometheus-kube-prometheus-alertmanager   ClusterIP   10.98.95.240     <none>        9093\/TCP   3m35s\nservice\/prometheus-kube-prometheus-operator       ClusterIP   10.101.213.133   <none>        443\/TCP    3m35s\nservice\/prometheus-kube-prometheus-prometheus     ClusterIP   10.110.12.91     <none>        9090\/TCP   3m35s\nservice\/prometheus-kube-state-metrics             ClusterIP   10.98.72.100     <none>        8080\/TCP   3m35s\nservice\/prometheus-prometheus-node-exporter       ClusterIP   10.107.130.247   <none>        9100\/TCP   3m35s\n\nNAME                                                 DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE\ndaemonset.apps\/prometheus-prometheus-node-exporter   4         4         4       4            4           <none>          3m34s\n\nNAME                                                  READY   UP-TO-DATE   AVAILABLE   AGE\ndeployment.apps\/prometheus-kube-prometheus-operator   1\/1     1            1           3m34s\ndeployment.apps\/prometheus-kube-state-metrics         1\/1     1            1           3m34s\n\nNAME                                                             DESIRED   CURRENT   READY   AGE\nreplicaset.apps\/prometheus-kube-prometheus-operator-54cfc96db7   1         1         1       3m34s\nreplicaset.apps\/prometheus-kube-state-metrics-5f5f8b8fdd         1         1         1       3m34s\n\nNAME                                                                    READY   AGE\nstatefulset.apps\/alertmanager-prometheus-kube-prometheus-alertmanager   1\/1     3m21s\nstatefulset.apps\/prometheus-prometheus-kube-prometheus-prometheus       1\/1     3m21s\n<\/code><\/pre>\n\n\n\n<p>Some resources do not have the <code>all<\/code> resource type. In this case, you can list individual resource types e.g;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl get pods -n &lt;namespace&gt;<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl get svc -n &lt;namespace&gt;<\/code><\/pre>\n\n\n\n<p>You can also check the config maps related to Prometheus using the command below. ConfigMaps are used to store and manage non-confidential configuration data for your applications<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl get configmaps --selector release=prometheus<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>\nNAME                                                           DATA   AGE\nprometheus-kube-prometheus-alertmanager-overview               1      6m3s\nprometheus-kube-prometheus-apiserver                           1      6m3s\nprometheus-kube-prometheus-cluster-total                       1      6m3s\nprometheus-kube-prometheus-controller-manager                  1      6m3s\nprometheus-kube-prometheus-etcd                                1      6m3s\nprometheus-kube-prometheus-grafana-datasource                  1      6m3s\nprometheus-kube-prometheus-grafana-overview                    1      6m3s\nprometheus-kube-prometheus-k8s-coredns                         1      6m3s\nprometheus-kube-prometheus-k8s-resources-cluster               1      6m3s\nprometheus-kube-prometheus-k8s-resources-multicluster          1      6m3s\nprometheus-kube-prometheus-k8s-resources-namespace             1      6m3s\nprometheus-kube-prometheus-k8s-resources-node                  1      6m3s\nprometheus-kube-prometheus-k8s-resources-pod                   1      6m3s\nprometheus-kube-prometheus-k8s-resources-workload              1      6m3s\nprometheus-kube-prometheus-k8s-resources-workloads-namespace   1      6m3s\nprometheus-kube-prometheus-kubelet                             1      6m3s\nprometheus-kube-prometheus-namespace-by-pod                    1      6m3s\nprometheus-kube-prometheus-namespace-by-workload               1      6m3s\nprometheus-kube-prometheus-node-cluster-rsrc-use               1      6m3s\nprometheus-kube-prometheus-node-rsrc-use                       1      6m3s\nprometheus-kube-prometheus-nodes                               1      6m3s\nprometheus-kube-prometheus-nodes-darwin                        1      6m3s\nprometheus-kube-prometheus-persistentvolumesusage              1      6m3s\nprometheus-kube-prometheus-pod-total                           1      6m3s\nprometheus-kube-prometheus-prometheus                          1      6m3s\nprometheus-kube-prometheus-proxy                               1      6m3s\nprometheus-kube-prometheus-scheduler                           1      6m3s\nprometheus-kube-prometheus-workload-total                      1      6m3s\n<\/code><\/pre>\n\n\n\n<p>There are also service monitors used by Prometheus operators to define the scraping configuration required to monitor services and endpoints within your Kubernetes cluster.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl get servicemonitor --selector release=prometheus<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>\nNAME                                                 AGE\nprometheus-kube-prometheus-alertmanager              13m\nprometheus-kube-prometheus-apiserver                 13m\nprometheus-kube-prometheus-coredns                   13m\nprometheus-kube-prometheus-kube-controller-manager   13m\nprometheus-kube-prometheus-kube-etcd                 13m\nprometheus-kube-prometheus-kube-proxy                13m\nprometheus-kube-prometheus-kube-scheduler            13m\nprometheus-kube-prometheus-kubelet                   13m\nprometheus-kube-prometheus-operator                  13m\nprometheus-kube-prometheus-prometheus                13m\nprometheus-kube-state-metrics                        13m\nprometheus-prometheus-node-exporter                  13m\n<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"accessing-prometheus-outside-k-8-s-cluster\">Accessing Prometheus Outside K8S cluster<\/h3>\n\n\n\n<p>You can check services related to Prometheus on the default namespace;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl get svc --selector release=prometheus<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>\nNAME                                      TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE\nprometheus-kube-prometheus-alertmanager   ClusterIP   10.98.95.240     <none>        9093\/TCP   10m\nprometheus-kube-prometheus-operator       ClusterIP   10.101.213.133   <none>        443\/TCP    10m\nprometheus-kube-prometheus-prometheus     ClusterIP   10.110.12.91     <none>        9090\/TCP   10m\nprometheus-kube-state-metrics             ClusterIP   10.98.72.100     <none>        8080\/TCP   10m\nprometheus-prometheus-node-exporter       ClusterIP   10.107.130.247   <none>        9100\/TCP   10m\n<\/code><\/pre>\n\n\n\n<p>As you can see, the Prometheus services are only meant for internal access within the cluster as depicted by the service type <strong><code>ClusterIP<\/code><\/strong>.<\/p>\n\n\n\n<p>Check for example, Prometheus endpoint on port 9090\/tcp, <strong>prometheus-kube-prometheus-prometheus<\/strong>, this service exposes Prometheus on internal cluster IP address.<\/p>\n\n\n\n<p>To be able to access Prometheus from outside the cluster, we need to change service type to NodePort. This exposes the service on a static port on each selected node in the cluster and the service becomes accessible on each node&#8217;s IP address and the static port.<\/p>\n\n\n\n<p>Thus, edit the service;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl edit service prometheus-kube-prometheus-prometheus<\/code><\/pre>\n\n\n\n<p>By default, this is how the service manifest looks like;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>\n# Please edit the object below. Lines beginning with a '#' will be ignored,\n# and an empty file will abort the edit. If an error occurs while saving this file will be\n# reopened with the relevant failures.\n#\napiVersion: v1\nkind: Service\nmetadata:\n  annotations:\n    meta.helm.sh\/release-name: prometheus\n    meta.helm.sh\/release-namespace: default\n  creationTimestamp: \"2023-05-23T19:30:49Z\"\n  labels:\n    app: kube-prometheus-stack-prometheus\n    app.kubernetes.io\/instance: prometheus\n    app.kubernetes.io\/managed-by: Helm\n    app.kubernetes.io\/part-of: kube-prometheus-stack\n    app.kubernetes.io\/version: 46.1.0\n    chart: kube-prometheus-stack-46.1.0\n    heritage: Helm\n    release: prometheus\n    self-monitor: \"true\"\n  name: prometheus-kube-prometheus-prometheus\n  namespace: default\n  resourceVersion: \"124975\"\n  uid: 177fb969-6d22-46f5-8e39-0b3c451b4da2\nspec:\n  clusterIP: 10.108.24.96\n  clusterIPs:\n  - 10.108.24.96\n  internalTrafficPolicy: Cluster\n  ipFamilies:\n  - IPv4\n  ipFamilyPolicy: SingleStack\n  ports:\n  - name: http-web\n    port: 9090\n    protocol: TCP\n    targetPort: 9090\n  selector:\n    app.kubernetes.io\/name: prometheus\n    prometheus: prometheus-kube-prometheus-prometheus\n  sessionAffinity: None\n  type: ClusterIP\nstatus:\n  loadBalancer: {}\n<\/code><\/pre>\n\n\n\n<p>We will edit this file and change <strong><code>type: ClusterIP<\/code><\/strong> to <strong><code>type: NodePort<\/code><\/strong>. Also, we will bind it to static NodePort that is currently not being used, <strong><code>30002<\/code><\/strong>.<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>\n  ports:\n  - name: http-web\n    port: 9090\n    protocol: TCP\n    targetPort: 9090\n<strong>    nodePort: 30002<\/strong>\n  selector:\n    app.kubernetes.io\/name: prometheus\n    prometheus: prometheus-kube-prometheus-prometheus\n  sessionAffinity: None\n  <strong>type: NodePort<\/strong>\n<\/code><\/pre>\n\n\n\n<p>Save and exit the file.<\/p>\n\n\n\n<p>Confirm the changes;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl get svc prometheus-kube-prometheus-prometheus<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>NAME                                    TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE\nprometheus-kube-prometheus-prometheus   NodePort   10.108.24.96   &lt;none&gt;        <strong>9090:30002\/TCP<\/strong>   12m<\/code><\/pre>\n\n\n\n<p>You should now be able to access Prometheus interface on any node IP via the address <strong><code>http:\/\/&lt;NodeIP&gt;:30002<\/code><\/strong>.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1892\" height=\"855\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/05\/kubernetes-prometheus-web-interface.png\" alt=\"\" class=\"wp-image-16818\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/05\/kubernetes-prometheus-web-interface.png?v=1684876522 1892w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/05\/kubernetes-prometheus-web-interface-768x347.png?v=1684876522 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/05\/kubernetes-prometheus-web-interface-1536x694.png?v=1684876522 1536w\" sizes=\"(max-width: 1892px) 100vw, 1892px\" \/><\/figure>\n\n\n\n<p>You may realize that some endpoints metrics are giving connection refused errors;<\/p>\n\n\n\n<p>In order to resolve this, we need to change the bind addresses for <strong><code>kube-controller-manager<\/code><\/strong>, <strong><code>etcd<\/code><\/strong>, <strong><code>kube-scheduler<\/code><\/strong> and <strong><code>kube-proxy<\/code><\/strong>. <\/p>\n\n\n\n<p>They all listen on loopback interfaces by default.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ss -altnp | grep -E  \"10257|2381|10249|10259\"<\/code><\/pre>\n\n\n\n<p>We will update the configurations and bind the address to 0.0.0.0. Please note this opens up access to these services from any interface. Be sure to setup proper firewall rules in place to prevent unauthorized access.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Update Kube Proxy bind address (<strong><code>cm<\/code><\/strong> is short form of <strong><code>configmaps<\/code><\/strong>)<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl edit cm kube-proxy -n kube-system<\/code><\/pre>\n\n\n\n<p>Under the section, <strong><code>config.conf: |-<\/code><\/strong>, change the <code>metricsBindAddress: \"\"<\/code> to <strong><code>metricsBindAddress: \"0.0.0.0:10249\"<\/code><\/strong><\/p>\n\n\n\n<p>Save and exit.<\/p>\n\n\n\n<p>To apply the changes, delete all kube-proxy pods to recreate new ones with updated bind address;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl delete pods -l k8s-app=kube-proxy -n kube-system<\/code><\/pre>\n\n\n\n<p>Check if Kube proxy pods have been recreated;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl get pod -l k8s-app=kube-proxy -n kube-system<\/code><\/pre>\n\n\n\n<p>check ports;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ss -altnp | grep 10249<\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Change ETCD Metrics bind address<\/li>\n<\/ul>\n\n\n\n<p>Edit the configuration file used by etcd and change the metrics bind address as follows;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo vim \/etc\/kubernetes\/manifests\/etcd.yaml<\/code><\/pre>\n\n\n\n<p>Change the line;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>--listen-metrics-urls=http:\/\/127.0.0.1:2381<\/code><\/pre>\n\n\n\n<p>to;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>--listen-metrics-urls=http:\/\/0.0.0.0:2381<\/code><\/pre>\n\n\n\n<p>Save and exit the file.<\/p>\n\n\n\n<p>Relevant pods will automatically restart&nbsp;and set the bind address to 0.0.0.0.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Change Kube Scheduler bind address<\/li>\n<\/ul>\n\n\n\n<p>Edit the configuration file used by scheduler and change the bind address as follows;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo vim \/etc\/kubernetes\/manifests\/kube-scheduler.yaml<\/code><\/pre>\n\n\n\n<p>Change the line;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>--bind-address=127.0.0.1<\/code><\/pre>\n\n\n\n<p>to;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>--bind-address=0.0.0.0<\/code><\/pre>\n\n\n\n<p>Save and exit the file.<\/p>\n\n\n\n<p>Relevant pods will automatically restart&nbsp;and set the bind address to 0.0.0.0.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Change Kube controller manager bind address;<\/li>\n<\/ul>\n\n\n\n<p>Edit the manifest configuration file used by controller manager and change the bind address as follows;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo vim \/etc\/kubernetes\/manifests\/kube-controller-manager.yaml<\/code><\/pre>\n\n\n\n<p>Change the line;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>--bind-address=127.0.0.1<\/code><\/pre>\n\n\n\n<p>to;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>--bind-address=127.0.0.1<\/code><\/pre>\n\n\n\n<p>Save and exit the file.<\/p>\n\n\n\n<p>Relevant pods will automatically restart&nbsp;and set the bind address to 0.0.0.0.<\/p>\n\n\n\n<p>Confirm the ports;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo lsof -i :2381,10249,10257,10259<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>\nCOMMAND    PID USER   FD   TYPE DEVICE SIZE\/OFF NODE NAME\nkube-sche 1239 root    3u  IPv6  22234      0t0  TCP *:10259 (LISTEN)\netcd      1245 root   14u  IPv6  22268      0t0  TCP *:2381 (LISTEN)\nkube-cont 1261 root    3u  IPv6  22936      0t0  TCP *:10257 (LISTEN)\nkube-prox 1866 root   11u  IPv6  24294      0t0  TCP *:10249 (LISTEN)\n<\/code><\/pre>\n\n\n\n<p>This should resolve issue with kube-prometheus-stack connection refused.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"accessing-grafana-outside-k-8-s-cluster\">Accessing Grafana Outside K8S Cluster<\/h3>\n\n\n\n<p>The kube-prometheus-stack helm chart installed, also install Grafana. The Grafana is can only be accessed within the cluster via port 80\/TCP.<\/p>\n\n\n\n<p>In order to access it externally, we will edit the service and change service type to NodePort as we did above for Prometheus.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl edit svc prometheus-grafana<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>\nspec:\nspec:\n  clusterIP: 10.111.147.239\n  clusterIPs:\n  - 10.111.147.239\n  externalTrafficPolicy: Cluster\n  internalTrafficPolicy: Cluster\n  ipFamilies:\n  - IPv4\n  ipFamilyPolicy: SingleStack\n  ports:\n  - name: service\n    port: 80\n    protocol: TCP\n    targetPort: 3000\n    <strong>nodePort: 30003<\/strong>\n  selector:\n    app.kubernetes.io\/instance: grafana\n    app.kubernetes.io\/name: grafana\n  sessionAffinity: None\n    #type: ClusterIP\n  <strong>type: NodePort<\/strong>\n..\n<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl get svc<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>NAME                                      TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE\n...\ngrafana                                   NodePort    10.111.147.239    &lt;none&gt;        80:30003\/TCP                 21m\n...<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"login-to-grafana-web-interface\">Login to Grafana Web Interface<\/h3>\n\n\n\n<p>You should now be able to Access Grafana outside the cluster on any cluster node&#8217;s IP on port 30003.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1448\" height=\"961\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/05\/grafana-web-ui-login-page.png\" alt=\"\" class=\"wp-image-16820\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/05\/grafana-web-ui-login-page.png?v=1684876589 1448w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/05\/grafana-web-ui-login-page-768x510.png?v=1684876589 768w\" sizes=\"(max-width: 1448px) 100vw, 1448px\" \/><\/figure>\n\n\n\n<p>You can generate Grafana admin user password by running the command below;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl get secret prometheus-grafana \\\n-o jsonpath=\"{.data.admin-password}\" | base64 --decode ; echo<\/code><\/pre>\n\n\n\n<p>Sample output;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>prom-operator<\/code><\/pre>\n\n\n\n<p>The default dashboard;<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1885\" height=\"842\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/05\/kubernetes-grafana-dashboard.png\" alt=\"\" class=\"wp-image-16821\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/05\/kubernetes-grafana-dashboard.png?v=1684876700 1885w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/05\/kubernetes-grafana-dashboard-768x343.png?v=1684876700 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/05\/kubernetes-grafana-dashboard-1536x686.png?v=1684876700 1536w\" sizes=\"(max-width: 1885px) 100vw, 1885px\" \/><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"grafana-prometheus-datasource\">Grafana Prometheus Datasource<\/h3>\n\n\n\n<p>The stack already preconfigured. Prometheus data source has already been added.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1896\" height=\"421\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/05\/kubernetes-prometheus-grafana-datasource.png\" alt=\"\" class=\"wp-image-16824\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/05\/kubernetes-prometheus-grafana-datasource.png?v=1684920015 1896w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/05\/kubernetes-prometheus-grafana-datasource-768x171.png?v=1684920015 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/05\/kubernetes-prometheus-grafana-datasource-1536x341.png?v=1684920015 1536w\" sizes=\"(max-width: 1896px) 100vw, 1896px\" \/><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"grafana-kubernetes-dashboards\">Grafana Kubernetes Dashboards<\/h3>\n\n\n\n<p>The stack also comes with some dashboard preconfigured.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1890\" height=\"901\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/05\/grafana-Kubernetes-dashboard.png\" alt=\"Monitoring Kubernetes with Prometheus and Grafana\" class=\"wp-image-16825\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/05\/grafana-Kubernetes-dashboard.png?v=1684920367 1890w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/05\/grafana-Kubernetes-dashboard-768x366.png?v=1684920367 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/05\/grafana-Kubernetes-dashboard-1536x732.png?v=1684920367 1536w\" sizes=\"(max-width: 1890px) 100vw, 1890px\" \/><\/figure>\n\n\n\n<p>Let&#8217;s check some dashboard;<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1893\" height=\"941\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/05\/kubernetes-cluster-dashboard-metrics.png\" alt=\"Kubernetes Monitoring with Prometheus and Grafana\" class=\"wp-image-16827\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/05\/kubernetes-cluster-dashboard-metrics.png?v=1684920721 1893w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/05\/kubernetes-cluster-dashboard-metrics-768x382.png?v=1684920721 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/05\/kubernetes-cluster-dashboard-metrics-1536x764.png?v=1684920721 1536w\" sizes=\"(max-width: 1893px) 100vw, 1893px\" \/><\/figure>\n\n\n\n<p>Explore other dashboards.<\/p>\n\n\n\n<p>Update everything to suite your needs!<\/p>\n\n\n\n<p>That concludes our guide on monitoring Kubernetes with Prometheus and Grafana.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"other-tutorials\">Other Tutorials<\/h2>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/monitor-docker-swarm-and-container-metrics-using-metricbeat\/\" target=\"_blank\" rel=\"noreferrer noopener\">Monitor Docker Swarm and Container metrics using Metricbeat<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/monitor-docker-swarm-node-metrics-using-grafana\/\" target=\"_blank\" rel=\"noreferrer noopener\">Monitor Docker Swarm Node Metrics using Grafana<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>In this tutorial, we will provide you with a step by step guide on Kubernetes monitoring with Prometheus and Grafana. Being able to monitor a<\/p>\n","protected":false},"author":10,"featured_media":16827,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"rank_math_lock_modified_date":false,"footnotes":""},"categories":[121],"tags":[6789,6788,6787,6783,6784,6786,6791,6790],"class_list":["post-16802","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-howtos","tag-install-grafana-on-kubernetes-cluster","tag-install-prometheus-on-kubernetes-cluster","tag-kube-prometheus-stack-controller-manager-connection-refused","tag-kube-prometheus-stack-etcd-connection-refused","tag-kube-prometheus-stack-kube-proxy-connection-refused","tag-kube-prometheus-stack-scheduler-connection-refused","tag-kubernetes-monitoring-with-prometheus-and-grafana","tag-monitor-kubernetes-cluster-with-prometheus-and-grafana","generate-columns","tablet-grid-50","mobile-grid-100","grid-parent","grid-50","resize-featured-image"],"_links":{"self":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/16802"}],"collection":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/users\/10"}],"replies":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/comments?post=16802"}],"version-history":[{"count":15,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/16802\/revisions"}],"predecessor-version":[{"id":20739,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/16802\/revisions\/20739"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media\/16827"}],"wp:attachment":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media?parent=16802"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/categories?post=16802"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/tags?post=16802"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}