{"id":13103,"date":"2022-06-08T23:29:03","date_gmt":"2022-06-08T20:29:03","guid":{"rendered":"https:\/\/kifarunix.com\/?p=13103"},"modified":"2024-03-09T15:22:12","modified_gmt":"2024-03-09T12:22:12","slug":"how-to-fix-filebeat-glibc-related-errors","status":"publish","type":"post","link":"https:\/\/kifarunix.com\/how-to-fix-filebeat-glibc-related-errors\/","title":{"rendered":"How to Fix Filebeat Glibc Related Errors on Ubuntu 22.04"},"content":{"rendered":"\n<p>In this tutorial, you will learn how to fix Filebeat Glibc related errors on Ubuntu 22.04 that is affecting users using glibc &gt;= 2.35.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ldd --version<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code><strong>ldd (Ubuntu GLIBC 2.35-0ubuntu3) 2.35<\/strong>\nCopyright (C) 2022 Free Software Foundation, Inc.\nThis is free software; see the source for copying conditions.  There is NO\nwarranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.\nWritten by Roland McGrath and Ulrich Drepper.<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">How to Fix Filebeat Glibc Related Errors on Ubuntu 22.04<\/h2>\n\n\n\n<p>Have you installed Filebeat on Ubuntu 22.04 but realized that it cannot run due to some Glibc related errors?<\/p>\n\n\n\n<p>In my demo setup, I have Elastic Stack 7.17.0 running on Debian 11.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>curl localhost:9200<\/code><\/pre>\n\n\n\n<p>Sample output <strong>&#8220;number&#8221; : &#8220;7.17.0&#8221;<\/strong>;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>\n{\n  \"name\" : \"debian11\",\n  \"cluster_name\" : \"elasticsearch\",\n  \"cluster_uuid\" : \"_JanDf4yRcCxVBnLgN0a5A\",\n  \"version\" : {\n    <strong>\"number\" : \"7.17.0\"<\/strong>,\n    \"build_flavor\" : \"default\",\n    \"build_type\" : \"deb\",\n    \"build_hash\" : \"bee86328705acaa9a6daede7140defd4d9ec56bd\",\n    \"build_date\" : \"2022-01-28T08:36:04.875279988Z\",\n    \"build_snapshot\" : false,\n    \"lucene_version\" : \"8.11.1\",\n    \"minimum_wire_compatibility_version\" : \"6.8.0\",\n    \"minimum_index_compatibility_version\" : \"6.0.0-beta1\"\n  },\n  \"tagline\" : \"You Know, for Search\"\n}\n<\/code><\/pre>\n\n\n\n<p>It is always recommended to install similar versions of all components. That is to say, if you are running Elastic Stack v7.17.0, then Elasticsearch, Kibana, Logstash and Beasts should all be of the same versions.<\/p>\n\n\n\n<p>Having said that, we installed Filebeat 7.17.0 and other Filebeat versions on Ubuntu 22.04 trying to check if the issue would go away. However, the same errors were experienced with some versions of Filebeat!<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>hostnamectl<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>\n   Static hostname: jellyfish\n         Icon name: computer-convertible\n           Chassis: convertible\n        Machine ID: a892921910db4c7aa544a53d6f775666\n           Boot ID: af6f2a87fb394cada730d00fbd56c9b1\n  <strong>Operating System: Ubuntu 22.04 LTS<\/strong>\n            Kernel: Linux 5.15.0-27-generic\n      Architecture: x86-64\n<\/code><\/pre>\n\n\n\n<p>Sample Glibc Errors experienced on Ubuntu 22.04 with different versions of Filebeat;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Fatal glibc error: rseq registration failed<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>\n...\n2022-06-08T17:13:39.465Z\tINFO\tinstance\/beat.go:328\tSetup Beat: filebeat; Version: 7.17.0\n2022-06-08T17:13:39.465Z\tINFO\t[index-management]\tidxmgmt\/std.go:184\tSet output.elasticsearch.index to 'filebeat-7.17.0' as ILM is enabled.\n2022-06-08T17:13:39.465Z\tINFO\t[esclientleg]\teslegclient\/connection.go:105\telasticsearch url: http:\/\/192.168.58.22:9200\n2022-06-08T17:13:39.467Z\tINFO\t[publisher]\tpipeline\/module.go:113\tBeat name: ceph-admin\n2022-06-08T17:13:39.479Z\tINFO\t[monitoring]\tlog\/log.go:142\tStarting metrics logging every 30s\n<strong>Fatal glibc error: rseq registration failed\nAborted (core dumped)<\/strong>\n<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>\n...\n<strong>runtime\/cgo: pthread_create failed: Operation not permitted\nSIGABRT: abort<\/strong>\nPC=0x7f928b9c6a7c m=5 sigcode=18446744073709551610\n\ngoroutine 0 [idle]:\nruntime: unknown pc 0x7f928b9c6a7c\nstack: frame={sp:0x7f92634578a0, fp:0x0} stack=[0x7f9262c581e8,0x7f9263457de8)\n...\n<\/code><\/pre>\n\n\n\n<p>If you experienced these errors, how would you fix them?<\/p>\n\n\n\n<p>Well, according to <a href=\"https:\/\/discuss.elastic.co\/t\/filebeat-and-glibc-errors-on-ubuntu-22-04\/306653\" target=\"_blank\" rel=\"noreferrer noopener\">this<\/a> topic on Elastic Forum, <a href=\"https:\/\/discuss.elastic.co\/u\/cmacknz\" target=\"_blank\" rel=\"noreferrer noopener\">MacKenzie<\/a> mentioned that &#8220;<em>glibc &gt;= 2.35 added a new <strong>rseq<\/strong> syscall that is not in our default list of allowed syscalls<\/em>&#8220;<\/p>\n\n\n\n<p>He went ahead and suggested some fixes.<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li><a href=\"#use-filebeat-7.17.2\">Upgrade to Elastic 7.17.2 and use Filebeat 7.17.2<\/a><\/li>\n\n\n\n<li><a href=\"#allow-filebeat-rseq-syscall\">Customize allowed Filebeat Syscalls to include rseq syscall<\/a><\/li>\n<\/ol>\n\n\n\n<p><strong><code>rseq<\/code><\/strong>, an acronym for <a href=\"https:\/\/events19.linuxfoundation.org\/wp-content\/uploads\/2017\/12\/Improve-Linux-User-Space-Core-Libraries-with-Restartable-Sequences-Mathieu-Desnoyers-EfficiOS.pdf\" target=\"_blank\" rel=\"noreferrer noopener\">Restartable Seqeunces<\/a>, is a system call that provides synchronization mechanism for per-CPU data which super-fast update operations on per-cpu data in user-space.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"use-filebeat-7.17.2\">Install Filebeat 7.17.2<\/h3>\n\n\n\n<p>As already stated before, it is always recommended to install similar versions of all components. As such, MacKenzie suggested that one can upgrade their Elastic Stack version to 7.17.2 and use <a href=\"https:\/\/www.elastic.co\/guide\/en\/beats\/libbeat\/7.17\/release-notes-7.17.2.html\" target=\"_blank\" rel=\"noreferrer noopener\">beta release versions of Filebeat 7.17.2<\/a>, the first release which ships with the fix to errors experienced when glibc &gt;= 2.35 is used;<\/p>\n\n\n\n<p>We installed Filebeat 7.17.2, and it indeed fixed the issue;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>filebeat version<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>filebeat version <strong>7.17.2<\/strong> (amd64), libbeat 7.17.2 &#91;f6042bc3407cc10201cfd8c7574d8b0a88a699db built 2022-03-28 09:47:58 +0000 UTC]<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>filebeat -e<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>...\n<strong>INFO\t&#91;publisher_pipeline_output]\tpipeline\/output.go:151\tConnection to backoff(elasticsearch(http:\/\/x.x.x.x:9200)) established<\/strong><\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"allow-filebeat-rseq-syscall\">Customize Allowed Filebeat Syscalls<\/h3>\n\n\n\n<p>Use this solution if you cannot upgrade your Elastic Stack for one or the other reasons.<\/p>\n\n\n\n<p>Filebeat is setup to utilize <a href=\"https:\/\/www.elastic.co\/guide\/en\/beats\/filebeat\/current\/linux-seccomp.html#linux-seccomp\" target=\"_blank\" rel=\"noreferrer noopener\">Linux secure computing mode<\/a> (seccomp), which exposes only specific system calls to Filebeat program which thus <em>minimizes the impact of unknown vulnerabilities that might be found in the process.<\/em><\/p>\n\n\n\n<p>As MacKenzie mentioned, GLIBC &gt;= 2.35 added a new <strong><code>req syscall<\/code><\/strong> which is not exposed to Filebeat by default.<\/p>\n\n\n\n<p>Thus, in order to be able fix the Filebeat issue with Glibc without the need to upgrade your Elastic Stack, you need to configure Filebeat to allow this specific system call.<\/p>\n\n\n\n<p>On Linux, Filebeat is set to enable seccomp by default.<\/p>\n\n\n\n<p>To allow rseq syscall, edit the filebeat.yml configuration file and add the lines below;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code><strong>seccomp:\n  default_action: allow \n  syscalls:\n  - action: allow\n    names:\n    - rseq<\/strong><\/code><\/pre>\n\n\n\n<p>You can simply copy and paste the command below on the terminal to update the Filebeat configuration file with the content above;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>\ncat >> \/etc\/filebeat\/filebeat.yml &lt;&lt; 'EOL'\nseccomp:\n  default_action: allow \n  syscalls:\n  - action: allow\n    names:\n    - rseq\nEOL\n<\/code><\/pre>\n\n\n\n<p>Next, you can test the changes by running;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>filebeat -e<\/code><\/pre>\n\n\n\n<p>You Filebeat should successfully connect to Elasticsearch or whatever your stash output is.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Other Tutorials<\/h3>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/install-elk-stack-8-x-on-ubuntu\/\" target=\"_blank\" rel=\"noreferrer noopener\">Install ELK Stack 8.x on Ubuntu<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>In this tutorial, you will learn how to fix Filebeat Glibc related errors on Ubuntu 22.04 that is affecting users using glibc &gt;= 2.35. How<\/p>\n","protected":false},"author":1,"featured_media":13105,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"rank_math_lock_modified_date":false,"footnotes":""},"categories":[121],"tags":[5360,5362,5364,5365,5363,5366,5361,5359,5367],"class_list":["post-13103","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-howtos","tag-fatal-glibc-error-rseq-registration-failed","tag-filebeat-glibc-2-35-errors","tag-filebeat-linux-secure-computing-syscalls","tag-filebeat-rseq-syscall","tag-filebeat-seccomp","tag-glibc-2-35-filebeat-errors","tag-how-to-fix-filebeat-glibc-related-errors-on-ubuntu-22-04","tag-runtime-cgo-pthread_create-failed-operation-not-permitted-sigabrt-abort","tag-seccomp-allow-rseq","generate-columns","tablet-grid-50","mobile-grid-100","grid-parent","grid-50","resize-featured-image"],"_links":{"self":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/13103"}],"collection":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/comments?post=13103"}],"version-history":[{"count":5,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/13103\/revisions"}],"predecessor-version":[{"id":20573,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/13103\/revisions\/20573"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media\/13105"}],"wp:attachment":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media?parent=13103"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/categories?post=13103"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/tags?post=13103"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}