{"id":5228,"date":"2020-03-20T22:43:14","date_gmt":"2020-03-20T19:43:14","guid":{"rendered":"https:\/\/kifarunix.com\/?p=5228"},"modified":"2022-06-06T19:20:48","modified_gmt":"2022-06-06T16:20:48","slug":"install-and-setup-glusterfs-storage-cluster-on-centos-8","status":"publish","type":"post","link":"https:\/\/kifarunix.com\/install-and-setup-glusterfs-storage-cluster-on-centos-8\/","title":{"rendered":"Install and Setup GlusterFS Storage Cluster on CentOS 8"},"content":{"rendered":"\n
In this guide, we are going to learn how to install and setup GlusterFS storage cluster on CentOS 8. GlusterFS<\/a> is an opensource distributed and scalable network file system that clusters various disk storage resources into a single global namespace. It is suitable for data-intensive tasks such as cloud storage and media streaming. Some of the common features for GlusterFS include;<\/p>\n\n\n\n Ensure that you have at least three storage nodes. Check our deployment architecture below.<\/p>\n\n\n\n Also, ensure that you have an extract virtual disk apart from the root disk where the OS is installed. In our case above, we have an extra disk, If using hostnames, ensure that they are resolvable.<\/p>\n\n\n\n Ensure that time is tightly synchronized between the three nodes. If you need to setup your NTP server on a CentOS 8 system, simply follow the link below;<\/p>\n\n\n\n Setup NTP Server using Chrony on CentOS 8<\/a><\/p>\n\n\n\n On each GlusterFS node, format and create a file system on the storage drive. You can use any partitioning method.<\/p>\n\n\n\n Format and create a primary partition on the disk.<\/p>\n\n\n\n Create a Filesystem on the disk. XFS is used here.<\/p>\n\n\n\n On each node, create a mount point for the GlusterFS storage drive, here in called bricks. A brick is a basic unit of GlusterFS storage.<\/p>\n\n\n\n Mount the GlusterFS storage drive on the bricks directory created above.<\/p>\n\n\n\n To automount the drive on system boot, simply update the \/etc\/fstab configuration file as follows;<\/p>\n\n\n\n Check the mounting;<\/p>\n\n\n\n Now that the prerequisites part is done, proceed to install GlusterFS on all nodes. GlusterFS is not available on the default repos. Hence, run the command below to create the repo.<\/p>\n\n\n\n Enable PowerTools repos in order to be able to install other required GlusterFS dependencies.<\/p>\n\n\n\n Install GlusterFS server package on CentOS 8 by running the command below;<\/p>\n\n\n\n Start and enable GlusterFS daemon to run on system boot by executing the command below;<\/p>\n\n\n\n To check the status of GlusterFS daemon;<\/p>\n\n\n\n In order for the GlusterFS nodes to be able to communicate, you need to open certain GlusterFS ports or services on a firewall. The To configure GlusterFS storage cluster, you first need to create a To probe other nodes, eg gfs02 and gfs03 from gfs01<\/strong> in our case, run the command below. Replace the names of the nodes accordingly. You can probe other nodes from any node<\/strong>.<\/p>\n\n\n\n To verify cluster peering status, run the command below from any node. Eg from node 1, gfs01;<\/p>\n\n\n\n To list all nodes in a TSP, eg from GlusterFS node 02, gfs01;<\/p>\n\n\n\n You can be able to add more serves or remove servers from the GlusterFS storage pool;<\/p>\n\n\n\n For example, to add a new server, you need to probe it from a node already in the pool. Replace the server-name<\/strong> accordingly.<\/p>\n\n\n\n To detach a server from the pool;<\/p>\n\n\n\n A GlusterFS storage volume can be created from the bricks that were created above. In this case, we have created bricks mounted on There are different configuration settings to consider while creating GlusterFS storage volume depending on various operational needs.<\/p>\n\n\n\n The GlusterFS volume can be created using the command;<\/p>\n\n\n\n Consult In this demo, we are going to learn how to setup replicated glusterfs storage volume.<\/p>\n\n\n\n Create a mount point for the volume on the brick on each cluster node.<\/strong> In our case, our brick is mounted on Next, created a replicated storage volume. For example, to create a replicated GlusterFS storage volume with three nodes called You can run the command from any node, replacing the names of the other peers and the associated bricks accordingly.<\/p>\n\n\n\n If everything goes well, you should get an output like;<\/p>\n\n\n\n After that, start the GlusterFS volume. Replace Once you have created and started the volumes, you can verify by running the command below from any node in the cluster.<\/p>\n\n\n\n There you go. You have successfully installed and setup replicated GlusterFS storage cluster.<\/p>\n\n\n\n Want to set a different type of GlusterFS volume? Consult GlusterFS documentation on setting up volume types<\/a>.<\/p>\n\n\n\n Learn how to setup GlusterFS distributed replicated storage volume by following the link below;<\/p>\n\n\n\n Setup GlusterFS Distributed Replicated Volume on CentOS 8<\/a><\/p>\n\n\n\n If anything, you can always check the log file, Now that you have a GlusterFS storage cluster running, you can setup a client to be able to utilize this storage. Gluster volumes can be accessed using the Gluster Native Client or via NFS v3.<\/p>\n\n\n\n In this demo, we are going to learn how to use the native GlusterFS client on CentOS 8.<\/p>\n\n\n\n Install the GlusterFS repositories on CentOS 8<\/p>\n\n\n\n Run system update;<\/p>\n\n\n\n Install Gluster Native Client on CentOS 8<\/p>\n\n\n\n To manually mount the Gluster storage volumes on a client, use the command;<\/p>\n\n\n\n Where:<\/p>\n\n\n\n Thus, to mount the Gluster storage volume from glusterfs node 01, gfs01<\/strong>, our mount command would be like;<\/p>\n\n\n\n Confirm the mounting;<\/p>\n\n\n\n To automatically mount the GlusterFS storage volume, you can update the Where In our next guide, we will discuss how to mount GlusterFS storage volume on a client using NFS.<\/p>\n\n\n\n That marks the end of our guide on how to install and setup GlusterFS storage cluster on CentOS 8.<\/p>\n\n\n\n GlusterFS Quick Start Guide<\/a><\/p>\n\n\n\n Setup Three Node Ceph Storage Cluster on Ubuntu 18.04<\/a><\/p>\n\n\n\n Install and Setup GlusterFS on Ubuntu 18.04<\/a><\/p>\n\n\n\n How to Configure NFS Server on Ubuntu 18.04 Server<\/a><\/p>\n\n\n\n How to Install and Configure NFS Server on RHEL\/CentOS 7<\/a><\/p>\n\n\n\nInstall and Setup GlusterFS Storage Cluster on CentOS 8<\/h2>\n\n\n\n
Prerequisites<\/h3>\n\n\n\n
<\/figure><\/div>\n\n\n\n
\/dev\/sdb<\/strong><\/code>.<\/p>\n\n\n\n
lsblk<\/code><\/pre>\n\n\n\n
\nNAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT\nsda 8:0 0 8G 0 disk \n\u251c\u2500sda1 8:1 0 1G 0 part \/boot\n\u2514\u2500sda2 8:2 0 7G 0 part \n \u251c\u2500cl-root 253:0 0 6.2G 0 lvm \/\n \u2514\u2500cl-swap 253:1 0 820M 0 lvm [SWAP]\nsdb 8:16 0 2G 0 disk \n\u2514\u2500sdb1 8:17 0 2G 0 part<\/strong> \nsr0 11:0 1 1024M 0 rom\n<\/code><\/pre>\n\n\n\n
Partition and Format the GlusterFS Storage Drives<\/h3>\n\n\n\n
parted \/dev\/sdb mklabel msdos<\/code><\/pre>\n\n\n\n
parted -a opt \/dev\/sdb mkpart primary xfs 0% 100%<\/code><\/pre>\n\n\n\n
mkfs.xfs \/dev\/sdb1 -L gfsdisk<\/code><\/pre>\n\n\n\n
Mount GlusterFS Storage Drives (Bricks)<\/h3>\n\n\n\n
mkdir -p \/export\/gfsbrick<\/code><\/pre>\n\n\n\n
mount \/dev\/sdb1 \/export\/gfsbrick\/<\/code><\/pre>\n\n\n\n
echo \"\/dev\/sdb1 \/export\/gfsbrick\/ xfs defaults 1 2\" >> \/etc\/fstab<\/code><\/pre>\n\n\n\n
df -hTP \/dev\/sdb1<\/code><\/pre>\n\n\n\n
Filesystem Type Size Used Avail Use% Mounted on\n\/dev\/sdb1 xfs 2.0G 47M 2.0G 3% \/export\/gfsbrick<\/code><\/pre>\n\n\n\n
Install GlusterFS 7 on CentOS 8<\/a><\/h3>\n\n\n\n
Install GlusterFS Repo<\/h4>\n\n\n\n
dnf install centos-release-gluster<\/code><\/pre>\n\n\n\n
dnf config-manager --set-enabled PowerTools<\/code><\/pre>\n\n\n\n
dnf install glusterfs-server<\/code><\/pre>\n\n\n\n
Running GlusterFS on CentOS 8<\/h4>\n\n\n\n
systemctl enable --now glusterd<\/code><\/pre>\n\n\n\n
systemctl status glusterd<\/code><\/pre>\n\n\n\n
\n\u25cf glusterd.service - GlusterFS, a clustered file-system server\n Loaded: loaded (\/usr\/lib\/systemd\/system\/glusterd.service; enabled; vendor preset: disabled)\n Active: active (running) since Fri 2020-03-20 19:51:37 EAT; 12s ago\n Docs: man:glusterd(8)\n Process: 2739 ExecStart=\/usr\/sbin\/glusterd -p \/var\/run\/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0\/SUCCESS)\n Main PID: 2740 (glusterd)\n Tasks: 9 (limit: 5047)\n Memory: 3.9M\n CGroup: \/system.slice\/glusterd.service\n \u2514\u25002740 \/usr\/sbin\/glusterd -p \/var\/run\/glusterd.pid --log-level INFO\n\nMar 20 19:51:37 gfs01.kifarunix-demo.com systemd[1]: Starting GlusterFS, a clustered file-system server...\nMar 20 19:51:37 gfs01.kifarunix-demo.com systemd[1]: Started GlusterFS, a clustered file-system server.\n<\/code><\/pre>\n\n\n\n
Allow GlusterFS on Firewall<\/h4>\n\n\n\n
24007-24008\/TCP<\/strong><\/code> are used for the communication between nodes, while
24009-24108\/TCP<\/strong><\/code> are required for client communication. To make this easy, simply use the service,
glusterfs<\/code><\/strong>.<\/p>\n\n\n\n
firewall-cmd --add-service=glusterfs --permanent<\/code><\/pre>\n\n\n\n
firewall-cmd --reload<\/code><\/pre>\n\n\n\n
Configure GlusterFS Trusted Storage Pool<\/h4>\n\n\n\n
trusted storage pool (TSP)<\/code><\/strong> which basically is a network of storage servers. TSP can be created by adding storage nodes to the TSP using the
gluster peer probe<\/strong><\/code> command as shown below.<\/p>\n\n\n\n
gluster peer probe gfs02<\/code><\/pre>\n\n\n\n
gluster peer probe gfs03<\/code><\/pre>\n\n\n\n
gluster peer status<\/code><\/pre>\n\n\n\n
\nNumber of Peers: 2\n\nHostname: gfs01.kifarunix-demo.com\nUuid: b2003be4-4bc9-4125-a2a1-0d46e4b2476f\nState: Peer in Cluster (Connected)\n\nHostname: gfs02\nUuid: 6bc3b730-b7bb-4d24-865a-c33a3db4489d\nState: Peer in Cluster (Connected)\n<\/code><\/pre>\n\n\n\n
gluster pool list<\/code><\/pre>\n\n\n\n
\nUUID\t\t\t\t\tHostname \tState\n6bc3b730-b7bb-4d24-865a-c33a3db4489d\tgfs02 \tConnected \n26cdb6a4-ad15-41e7-bb72-2da0a9906fa6\tgfs03 \tConnected \nb2003be4-4bc9-4125-a2a1-0d46e4b2476f\tlocalhost\tConnected\n<\/code><\/pre>\n\n\n\n
gluster peer probe server-name<\/strong><\/code><\/pre>\n\n\n\n
gluster peer detach server-name<\/strong><\/code><\/pre>\n\n\n\n
Setup GlusterFS Storage Volume<\/h4>\n\n\n\n
\/export\/gfsbrick<\/strong><\/code> on each node.<\/p>\n\n\n\n
GlusterFS Storage Volume Types<\/h4>\n\n\n\n
Distributed<\/code><\/strong>: Files are distributed across the bricks in the volume.<\/li>
Replicated<\/code><\/strong>: Files are replicated across the bricks in the volume. It ensures high storage availability and reliability.<\/li>
Distributed Replicated<\/code><\/strong>: Files are distributed across the replicated bricks in the volume. Ensures high-reliability, scalability and improved read performance.<\/li>
Arbitrated Replicated<\/code><\/strong>: Files are replicated across two bricks in a replica set and only the metadata is replicated to the third brick. Ensures data consistency.<\/li>
Dispersed<\/code><\/strong>: Files are dispersed across the bricks in the volume.<\/li>
Distributed Dispersed<\/code><\/strong>: Data is distributed across the dispersed sub-volume.<\/li><\/ul>\n\n\n\n
gluster volume create <NEW-VOLNAME> [stripe <COUNT>] [[replica <COUNT> [arbiter <COUNT>]]|[replica 2 thin-arbiter 1]] [disperse [<COUNT>]] [redundancy <COUNT>] [transport <tcp|rdma|tcp,rdma>] <NEW-BRICK> ...<TA-BRICK><\/code><\/pre>\n\n\n\n
man gluster<\/strong><\/code> for more details.<\/p>\n\n\n\n
Setup Replicated GlusterFS Storage Volume<\/a><\/h4>\n\n\n\n
\/export\/gfsbrick<\/code>.<\/p>\n\n\n\n
mkdir \/export\/gfsbrick\/gfsvol01<\/code><\/pre>\n\n\n\n
gfsvol01<\/strong><\/code>;<\/p>\n\n\n\n
gluster volume create gfsvol01 replica 3 transport tcp gfs01:\/export\/gfsbrick\/gfsvol01 gfs02:\/export\/gfsbrick\/gfsvol01 gfs03:\/export\/gfsbrick\/gfsvol01<\/code><\/pre>\n\n\n\n
volume create: gfsvol01: success: please start the volume to access data<\/strong><\/code><\/pre>\n\n\n\n
gfsvol01<\/strong><\/code> with the name of your volume.<\/p>\n\n\n\n
gluster volume start gfsvol01<\/strong><\/code><\/pre>\n\n\n\n
Verify GlusterFS Volumes<\/h4>\n\n\n\n
gluster volume info all<\/code><\/pre>\n\n\n\n
\nVolume Name: gfsvol01\nType: Replicate\nVolume ID: 9361067e-da82-411c-9c83-2ab1c2b888aa\nStatus: Started\nSnapshot Count: 0\nNumber of Bricks: 1 x 3 = 3\nTransport-type: tcp\nBricks:\nBrick1: gfs01:\/export\/gfsbrick\/gfsvol01\nBrick2: gfs02:\/export\/gfsbrick\/gfsvol01\nBrick3: gfs03:\/export\/gfsbrick\/gfsvol01\nOptions Reconfigured:\ntransport.address-family: inet\nstorage.fips-mode-rchecksum: on\nnfs.disable: on\nperformance.client-io-threads: off\n<\/code><\/pre>\n\n\n\n
\/var\/log\/glusterfs\/glusterd.log<\/code><\/strong> for any would be issues.<\/p>\n\n\n\n
Install and Setup GlusterFS client on CentOS 8<\/a><\/h3>\n\n\n\n
dnf install centos-release-gluster<\/code><\/pre>\n\n\n\n
dnf update<\/code><\/pre>\n\n\n\n
dnf install glusterfs glusterfs-fuse<\/code><\/pre>\n\n\n\n
Mounting Gluster Storage Volumes on Client<\/h3>\n\n\n\n
mount -t glusterfs HOSTNAME-OR-IPADDRESS:\/VOLNAME MOUNTDIR<\/code><\/pre>\n\n\n\n
-t glusterfs<\/code><\/strong> specifies the GlusterFS filesystem.<\/li>
HOSTNAME-OR-IPADDRESS<\/code><\/strong> is any gluster node eg, gfs01, gf02 or gfs03. Ensure hostnames are resolvable.<\/li>
\/VOLNAME<\/strong><\/code> is the name of the GlusterFS storage volume e.g
gfsvol01<\/strong><\/code> in this demo.<\/li>
MOUNTDIR<\/strong><\/code> is the mount point on the client.<\/li><\/ul>\n\n\n\n
mkdir \/media\/gfs<\/code><\/pre>\n\n\n\n
mount -t glusterfs gfs01:\/gfsvol01<\/strong> \/media\/gfs\/<\/code><\/pre>\n\n\n\n
df -hT<\/code><\/pre>\n\n\n\n
\nFilesystem Type Size Used Avail Use% Mounted on\ndevtmpfs devtmpfs 1.4G 0 1.4G 0% \/dev\ntmpfs tmpfs 1.4G 0 1.4G 0% \/dev\/shm\ntmpfs tmpfs 1.4G 9.9M 1.4G 1% \/run\ntmpfs tmpfs 1.4G 0 1.4G 0% \/sys\/fs\/cgroup\n\/dev\/mapper\/cl-root xfs 8.0G 5.7G 2.4G 71% \/\n\/dev\/sda1 ext4 976M 244M 666M 27% \/boot\n...\ngfs01:\/gfsvol01 fuse.glusterfs 2.0G 67M 2.0G 4% \/media\/gfs<\/strong>\n<\/code><\/pre>\n\n\n\n
\/etc\/fstab<\/strong><\/code> with the line;<\/p>\n\n\n\n
gfs01:\/gfsvol01<\/strong> \/media\/gfs\/ glusterfs defaults,_netdev 0 0<\/code><\/pre>\n\n\n\n
_netdev<\/code><\/strong> ensures that the filesystem is not mounted until when the network is enabled and running.<\/p>\n\n\n\n
Reference<\/h3>\n\n\n\n
Related Tutorials<\/h3>\n\n\n\n