{"id":7282,"date":"2020-11-29T07:59:26","date_gmt":"2020-11-29T04:59:26","guid":{"rendered":"https:\/\/kifarunix.com\/?p=7282"},"modified":"2024-03-14T23:38:46","modified_gmt":"2024-03-14T20:38:46","slug":"monitoring-gitlab-metrics-with-prometheus-and-grafana","status":"publish","type":"post","link":"https:\/\/kifarunix.com\/monitoring-gitlab-metrics-with-prometheus-and-grafana\/","title":{"rendered":"Monitoring Gitlab Metrics with Prometheus and Grafana"},"content":{"rendered":"\n

In this tutorial, we will learn about monitoring Gitlab metrics with Prometheus and Grafana. Grafana<\/a> is a data visualization and monitoring tool and supports time series datastores such as Graphite, InfluxDB, Prometheus, Elasticsearch. Prometheus<\/a> is a powerful time-series monitoring service, providing a flexible platform for monitoring GitLab and other software products. GitLab provides out of the box monitoring with Prometheus, providing easy access to high quality time-series monitoring of GitLab services.<\/em> Grafana can be used to achieve better visualization of the metrics collected by the Prometheus.<\/p>\n\n\n\n

Want more Grafana and Prometheus? Check the links below;<\/p>\n\n\n\n

Prometheus: Up & Running: Infrastructure and Application Performance Monitoring<\/a><\/p>\n\n\n\n

Learn Grafana 7.0: A beginner’s guide to getting well versed in analytics, interactive dashboards, and monitoring<\/a><\/p>\n\n\n\n

Using Prometheus and Grafana to Monitor Gitlab Metrics<\/h2>\n\n\n\n

Before you can proceed, we assume you have a Gitlab server up and running. However, you can follow the link below to install and setup Gitlab server on an Ubuntu 20.04 system;<\/p>\n\n\n\n

Install Gitlab with SSL\/TLS Certificate on Ubuntu 20.04<\/a><\/p>\n\n\n\n

By default, Omnibus Gitlab installations comes bundled with Grafana and Prometheus. To confirm this, simply check the status of Gitlab services;<\/p>\n\n\n\n

gitlab-ctl status<\/code><\/pre>\n\n\n\n
run: alertmanager: (pid 4248) 22043s; run: log: (pid 4158) 22046s\nrun: gitaly: (pid 4273) 22042s; run: log: (pid 4147) 22046s\nrun: gitlab-exporter: (pid 4285) 22042s; run: log: (pid 4151) 22046s\nrun: gitlab-workhorse: (pid 4297) 22042s; run: log: (pid 4157) 22046s\nrun: grafana: (pid 96055) 13581s; run: log: (pid 4152) 22046s\nrun: logrotate: (pid 190276) 422s; run: log: (pid 4156) 22046s\nrun: nginx: (pid 96045) 13583s; run: log: (pid 4155) 22047s\nrun: node-exporter: (pid 4352) 22041s; run: log: (pid 4148) 22047s\nrun: postgres-exporter: (pid 4384) 22041s; run: log: (pid 4159) 22047s\nrun: postgresql: (pid 4425) 22038s; run: log: (pid 4149) 22047s\nrun: prometheus: (pid 78644) 15085s; run: log: (pid 4160) 22047s\nrun: puma: (pid 95895) 13594s; run: log: (pid 4150) 22047s\nrun: redis: (pid 4469) 22036s; run: log: (pid 4153) 22047s\nrun: redis-exporter: (pid 4477) 22036s; run: log: (pid 4161) 22047s\nrun: sidekiq: (pid 95784) 13602s; run: log: (pid 4154) 22047s\n<\/code><\/pre>\n\n\n\n
As you can see, we have Grafana and Prometheus services up and running. Similarly, you can see that we have various exporters such as node-exporter<\/code><\/strong>, gitlab-exporter<\/code> for collecting host and Gitlab performance metrics.<\/p>\n\n\n\n
You can check the status of the individual service as follows;<\/p>\n\n\n\n
gitlab-ctl status prometheus<\/code><\/pre>\n\n\n\n
gitlab-ctl status grafana<\/code><\/pre>\n\n\n\n
Configuring Prometheus on Gitlab Server<\/h3>\n\n\n\n
By default, Prometheus is configured to be locally accessible from the Gitlab server itself. If you want to be able to access Prometheus externally, edit the configuration file and adjust the lines below accordingly.<\/p>\n\n\n\n
vim \/etc\/gitlab\/gitlab.rb<\/code><\/pre>\n\n\n\n
##! Advanced settings. Should be changed only if absolutely needed.\n#prometheus['listen_address'] = 'localhost:9090'\nprometheus['listen_address'] = '192.168.57.3:9090'\n<\/strong>#\n...<\/code><\/pre>\n\n\n\n
You can replace localhost with a FQDN or an IP address.<\/p>\n\n\n\n
If you do not want to allow external access to Gitlab Prometheus, then you can skip the above.<\/p>\n\n\n\n
Once you have made the changes, be sure to reconfigure Gitlab services for the changes to take effect;<\/p>\n\n\n\n
gitlab-ctl reconfigure<\/code><\/pre>\n\n\n\n
...\nRecipe: gitlab::gitlab-rails\n  * execute[clear the gitlab-rails cache] action run\n    - execute \/opt\/gitlab\/bin\/gitlab-rake cache:clear\nRecipe: monitoring::prometheus\n  * execute[reload prometheus] action run\n    - execute \/opt\/gitlab\/bin\/gitlab-ctl hup prometheus\n\nRunning handlers:\nRunning handlers complete\nChef Infra Client finished, 9\/811 resources updated in 01 minutes 09 seconds\ngitlab Reconfigured!\n<\/code><\/pre>\n\n\n\n
Viewing Gitlab Performance Metrics on Prometheus<\/h3>\n\n\n\n
You can access Prometheus dashboard, either locally or externally depending on your setup. In my case, we have configured Prometheus for external access.<\/p>\n\n\n\n
To allow external access, ensure that port 9090\/tcp<\/strong><\/code> is opened on firewall if firewall is running.<\/p>\n\n\n\n
Ubuntu systems and the likes;<\/p>\n\n\n\n
ufw allow 9090\/tcp<\/code><\/pre>\n\n\n\n
CentOS systems and the likes;<\/p>\n\n\n\n
firewall-cmd --add-port=9090\/tcp --permanent\nfirewall-cmd --reload<\/code><\/pre>\n\n\n\n
Then navigate to the Prometheus dashboard locally, (http:\/\/localhost:9090<\/strong><\/code>) or externally (http:\/\/fqdn-or-IP:9090<\/strong><\/code>).<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
There are a sample Prometheus dashboard queries you have been provided with on Gitlab documentation page<\/a>;<\/p>\n\n\n\n