{"id":13053,"date":"2022-06-03T23:45:04","date_gmt":"2022-06-03T20:45:04","guid":{"rendered":"https:\/\/kifarunix.com\/?p=13053"},"modified":"2024-03-09T15:45:54","modified_gmt":"2024-03-09T12:45:54","slug":"setup-replicated-glusterfs-volume-on-ubuntu","status":"publish","type":"post","link":"https:\/\/kifarunix.com\/setup-replicated-glusterfs-volume-on-ubuntu\/","title":{"rendered":"Setup Replicated GlusterFS Volume on Ubuntu"},"content":{"rendered":"\n<p>Follow through this tutorial to learn how to setup replicated GlusterFS volume on Ubuntu. There are&nbsp;<a href=\"https:\/\/docs.gluster.org\/en\/main\/Quick-Start-Guide\/Architecture\/\" target=\"_blank\" rel=\"noreferrer noopener\">different types of Volume architectures<\/a>&nbsp;that you may want to consider. These include;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><a href=\"https:\/\/kifarunix.com\/install-and-setup-glusterfs-on-ubuntu\/#distributed\">Distributed GlusterFS Volume<\/a><\/li>\n\n\n\n<li><a href=\"#replicated\">Replicated GlusterFS Volume<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/kifarunix.com\/setup-glusterfs-distributed-replicated-volume-on-centos-8\/\" target=\"_blank\" rel=\"noreferrer noopener\">Distributed Replicated GlusterFS Volume<\/a><\/li>\n\n\n\n<li>Striped GlusterFS Volume<\/li>\n\n\n\n<li>Distributed Striped GlusterFS Volume<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"replicated\">Setup Replicated GlusterFS Volume on Ubuntu<\/h3>\n\n\n\n<p>Replicated GlusterFS volume provides reliability and data redundancy and cushion against data loss in case one of the bricks get damaged. This is because exact copies of the data are stored on all the bricks that makes up the volume. You need <em>at least two bricks to create a volume with 2 replicas or a minimum of three bricks to create a volume of 3 replicas<\/em>.<\/p>\n\n\n\n<p>It is recommended to use at least 3 bricks in order to avoid the issue with <a href=\"https:\/\/docs.gluster.org\/en\/latest\/Administrator-Guide\/Split-brain-and-ways-to-deal-with-it\/\" target=\"_blank\" rel=\"noreferrer noopener\">split brain<\/a>.<\/p>\n\n\n\n<p>In our deployment architecture, we are using two storage servers with extra storage partition attached apart from the root partition.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Storage Node 1:\n<ul class=\"wp-block-list\">\n<li>Hostname: gfs01.kifarunix-demo.com<\/li>\n\n\n\n<li>IP address: 192.168.57.6<\/li>\n\n\n\n<li>Gluster Storage Disk: \/dev\/sdb1<\/li>\n\n\n\n<li>Size: 4GB<\/li>\n\n\n\n<li>Mount Point: \/gfsvolume<\/li>\n\n\n\n<li>OS: Ubuntu 22.04\/Ubuntu 20.04<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>Storage Node 2:\n<ul class=\"wp-block-list\">\n<li>Hostname: gfs02.kifarunix-demo.com<\/li>\n\n\n\n<li>IP address: 192.168.56.124<\/li>\n\n\n\n<li>Gluster Storage Disk: \/dev\/sdb1<\/li>\n\n\n\n<li>Size: 4GB<\/li>\n\n\n\n<li>Mount Point: \/gfsvolume<\/li>\n\n\n\n<li>OS: Ubuntu 22.04\/Ubuntu 20.04<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>Hostname: gfs03.kifarunix-demo.com\n<ul class=\"wp-block-list\">\n<li>IP address: 192.168.56.125<\/li>\n\n\n\n<li>Gluster Storage Disk: \/dev\/sdb1<\/li>\n\n\n\n<li>Size: 4GB<\/li>\n\n\n\n<li>Mount Point: \/gfsvolume<\/li>\n\n\n\n<li>OS: Ubuntu 22.04\/Ubuntu 20.04<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>GlusterFS Client:\n<ul class=\"wp-block-list\">\n<li>Hostname: gfsclient.kifarunix-demo.com<\/li>\n\n\n\n<li>IP address: 192.168.43.197<\/li>\n\n\n\n<li>OS: Ubuntu 22.04\/Ubuntu 20.04<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<p>Before you can proceed;<\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/install-and-setup-glusterfs-on-ubuntu\/#install-glusterfs\" target=\"_blank\" rel=\"noreferrer noopener\">Install GlusterFS Server on Ubuntu<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/install-and-setup-glusterfs-on-ubuntu\/#glusterfs-firewall\" target=\"_blank\" rel=\"noreferrer noopener\">Open GlusterFS Ports on Firewall<\/a><\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Create GlusterFS Trusted Storage Pool<\/h3>\n\n\n\n<p>A storage pool is a cluster of storage nodes which provides bricks to the storage volume.<\/p>\n\n\n\n<p>To create GlusterFS TSP, run the command below from either of the nodes, replacing <strong>SERVER<\/strong> with the other node being probed.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>gluster peer probe <em>SERVER<\/em><\/code><\/pre>\n\n\n\n<p>For example, to create the trusted storage pool containing Node 02 and Node 03 from Node 01;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>gluster peer probe gfs02<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>gluster peer probe gfs03<\/code><\/pre>\n\n\n\n<p>If all is well, you should get a successful probe; <strong><code>peer probe: success<\/code><\/strong>.<\/p>\n\n\n\n<p>To get the status of the TSP peers;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>gluster peer status<\/code><\/pre>\n\n\n\n<p>Sample output;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>Number of Peers: 2\n\nHostname: gfs02\nUuid: b81803a8-893a-499e-9a87-6bac00a62822\nState: Peer in Cluster (Connected)\n\nHostname: gfs03\nUuid: 88cf40a0-d458-4080-8c7a-c3cddbce86c0\nState: Peer in Cluster (Connected)\n<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Create Replicated GlusterFS Storage Volume<\/h3>\n\n\n\n<p>First of all ensure the storage drive is mounted. In our setup, we are using non root partition, <strong><code>\/dev\/sdb1<\/code><\/strong> mounted under <strong><code>\/gfsvolume<\/code><\/strong>.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>df -hTP \/gfsvolume\/<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>Filesystem     Type  Size  Used Avail Use% Mounted on\n\/dev\/sdb1      ext4  3.9G   24K  3.7G   1% \/gfsvolume<\/code><\/pre>\n\n\n\n<p>To create a replicated GlusterFS storage volume, use the <strong><code>gluster volume create<\/code><\/strong> command, whose CLI syntax is shown below;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>gluster volume create &lt;NEW-VOLNAME&gt; &#91;&#91;replica &lt;COUNT&gt; \\\n&#91;arbiter &lt;COUNT&gt;]]|&#91;replica 2 thin-arbiter 1]] &#91;disperse &#91;&lt;COUNT&gt;]] \\\n&#91;disperse-data &lt;COUNT&gt;] &#91;redundancy &lt;COUNT&gt;] &#91;transport &lt;tcp|rdma|tcp,rdma&gt;] \\\n&lt;NEW-BRICK&gt; &lt;TA-BRICK&gt;... &#91;force]<\/code><\/pre>\n\n\n\n<p>For example, to create a replicated storage volume using the two nodes, replace the name of the volume, <strong><code>replicated_volume<\/code><\/strong> as well as the nodes host-names accordingly;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>gluster volume create <strong>replicated_volume<\/strong> replica 3 transport tcp gfs01:\/gfsvolume\/gv0 \\\ngfs02:\/gfsvolume\/gv0 \\\ngfs03:\/gfsvolume\/gv0<\/code><\/pre>\n\n\n\n<p>Sample command output;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>volume create: replicated_volume: success: please start the volume to access data<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Start GlusterFS Volume<\/h3>\n\n\n\n<p>Once you have created the volume, you can now start it for you to start storing data in the volume.<\/p>\n\n\n\n<p>The command, <code>gluster volume start VOLUME_NAME<\/code>, can be used to start volume.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>gluster volume start replicated_volume<\/code><\/pre>\n\n\n\n<p>Sample command output;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>volume start: replicated_volume: success<\/code><\/pre>\n\n\n\n<p>Check the volume status;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>gluster volume status<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>Status of volume: replicated_volume\nGluster process                             TCP Port  RDMA Port  Online  Pid\n------------------------------------------------------------------------------\nBrick gfs01:\/gfsvolume\/gv0                  49152     0          Y       2050 \nBrick gfs02:\/gfsvolume\/gv0                  50073     0          Y       16260\nBrick gfs03:\/gfsvolume\/gv0                  60961     0          Y       1421 \nSelf-heal Daemon on localhost               N\/A       N\/A        Y       2071 \nSelf-heal Daemon on gfs03                   N\/A       N\/A        Y       1438 \nSelf-heal Daemon on gfs02                   N\/A       N\/A        Y       16277\n \nTask Status of Volume replicated_volume\n------------------------------------------------------------------------------\nThere are no active volume tasks\n<\/code><\/pre>\n\n\n\n<p>Check gluster information;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>gluster volume info<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code> \nVolume Name: replicated_volume\nType: Replicate\nVolume ID: 4f522843-13df-4042-b73a-ff6722dc9891\nStatus: Started\nSnapshot Count: 0\nNumber of Bricks: 1 x 3 = 3\nTransport-type: tcp\nBricks:\nBrick1: gfs01:\/gfsvolume\/gv0\nBrick2: gfs02:\/gfsvolume\/gv0\nBrick3: gfs03:\/gfsvolume\/gv0\nOptions Reconfigured:\ntransport.address-family: inet\nstorage.fips-mode-rchecksum: on\nnfs.disable: on\nperformance.client-io-threads: off\n<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Open GlusterFS Volumes Ports on Firewall<\/h3>\n\n\n\n<p>In order for the clients to connect to the volumes created, you need to open the respective node volume port on firewall. The ports are shown using the <strong><code>gluster volume status<\/code><\/strong> command above.<\/p>\n\n\n\n<p><strong>Similarly, ensure that the nodes can communicate to each other on these nodes<\/strong><\/p>\n\n\n\n<p>For example if your using UFW, on Node 01, allow clients and other Gluster nodes to connect to port 49152\/tcp by running the command below;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ufw allow from &lt;Client-IP-or-Network&gt; to any port 49152 proto tcp comment \"GlusterFS Client Access\"<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>ufw allow from &lt;Node02 IP&gt; to any port 49152 proto tcp comment \"GlusterFS Node02\"<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>ufw allow from &lt;Node03 IP&gt; to any port 49152 proto tcp comment \"GlusterFS Node03\"<\/code><\/pre>\n\n\n\n<p> On Node 02, allow clients to connect to port 50073\/tcp by running the command below;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ufw allow from &lt;Client-IP-or-Network&gt; to any port 50073 proto tcp comment \"GlusterFS Client Access\"<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>ufw allow from &lt;Node01-IP&gt; to any port 50073 proto tcp comment \"GlusterFS Node01\"<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>ufw allow from &lt;Node03 IP&gt; to any port 50073 proto tcp comment \"GlusterFS Node03\"<\/code><\/pre>\n\n\n\n<p>On Node 03, allow clients to connect to port 60961\/tcp by running the command below;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ufw allow from &lt;Client-IP-or-Network&gt; to any port 60961 proto tcp comment \"GlusterFS Client Access\"<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>ufw allow from &lt;Node01 IP&gt; to any port 60961 proto tcp comment \"GlusterFS Node01\"<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>ufw allow from &lt;Node02 IP&gt; to any port 60961 proto tcp comment \"GlusterFS Node02\"<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Mount Replicated GlusterFS Volume on Clients<\/h3>\n\n\n\n<p>As an example, we are using Ubuntu systems as GlusterFS clients.<\/p>\n\n\n\n<p>Thus, <a href=\"https:\/\/kifarunix.com\/install-and-setup-glusterfs-on-ubuntu\/#install-glusterfs-client\" target=\"_blank\" rel=\"noreferrer noopener\">install GlusterFS client<\/a> and proceed as follows to mount the replicated GlusterFS volume.<\/p>\n\n\n\n<p>Ensure the client can resolve the Gluster nodes hostnames.<\/p>\n\n\n\n<p>Create the mount point<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>mkdir \/mnt\/gfsvol<\/code><\/pre>\n\n\n\n<p>Mount the distributed volume. If using domain names, ensure they are resolvable.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>mount -t glusterfs gfs01:\/replicated_volume \/mnt\/gfsvol\/<\/code><\/pre>\n\n\n\n<p>Run the df command to check the mounted filesystems.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>df -hTP \/mnt\/gfsvol\/<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>Filesystem               Type            Size  Used Avail Use% Mounted on\ngfs01:\/replicated_volume fuse.glusterfs  3.9G   41M  3.7G   2% \/mnt\/gfsvol<\/code><\/pre>\n\n\n\n<p>From other clients, you can mount the volume on the other node;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>mount -t glusterfs gfs02:\/replicated_volume \/mnt\/gfsvol\/<\/code><\/pre>\n\n\n\n<p>To auto-mount the volume on system boot, you need to add the line below to <code>\/etc\/fstab<\/code>.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>gfs01:\/replicated_volume \/mnt\/gfsvol glusterfs defaults,_netdev 0 0<\/code><\/pre>\n\n\n\n<p>To test the data distribution, create two test files on the client. One of the file will be stored one of the volumes and the other file on the other volume. see example below;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>mkdir \/mnt\/gfsvol\/Test-dir\ntouch \/mnt\/gfsvol\/Test-dir\/{test-file,test-file-two}<\/code><\/pre>\n\n\n\n<p>If you can check on node01, node02 and node 03, they should all contain the same files<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>ls \/gfsvolume\/gv0\/Test-dir\/\ntest-file  test-file-two<\/code><\/pre>\n\n\n\n<p>On node02,<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>ls \/gfsvolume\/gv0\/Test-dir\/\ntest-file  test-file-two<\/code><\/pre>\n\n\n\n<p>On node03,<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>ls \/gfsvolume\/gv0\/Test-dir\/\ntest-file  test-file-two<\/code><\/pre>\n\n\n\n<p>That concludes our guide on setting up replicated GlusterFS volume on Ubuntu.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Other tutorials<\/h3>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/easily-install-and-configure-samba-file-server-on-ubuntu-22-04\/\" target=\"_blank\" rel=\"noreferrer noopener\">Easily Install and Configure Samba File Server on Ubuntu 22.04<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/install-couchbase-server-on-ubuntu\/\" target=\"_blank\" rel=\"noreferrer noopener\">Install Couchbase Server on Ubuntu 22.04\/Ubuntu 20.04<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/install-and-setup-ceph-storage-cluster-on-ubuntu\/\" target=\"_blank\" rel=\"noreferrer noopener\">Install and Setup Ceph Storage Cluster on Ubuntu 20.04<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Follow through this tutorial to learn how to setup replicated GlusterFS volume on Ubuntu. There are&nbsp;different types of Volume architectures&nbsp;that you may want to consider.<\/p>\n","protected":false},"author":1,"featured_media":13046,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"rank_math_lock_modified_date":false,"footnotes":""},"categories":[39,345,121],"tags":[5348,5350,5349,5346,5347,5345],"class_list":["post-13053","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-storage","category-glusterfs","category-howtos","tag-glustefs","tag-gluster-replica-3-bricks","tag-gluster-volume-create","tag-linux-glusterfs-replicated-volume","tag-replicated-volume-glusterfs","tag-setup-replicated-glusterfs-volume-ubuntu","generate-columns","tablet-grid-50","mobile-grid-100","grid-parent","grid-50","resize-featured-image"],"_links":{"self":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/13053"}],"collection":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/comments?post=13053"}],"version-history":[{"count":5,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/13053\/revisions"}],"predecessor-version":[{"id":20575,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/13053\/revisions\/20575"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media\/13046"}],"wp:attachment":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media?parent=13053"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/categories?post=13053"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/tags?post=13053"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}