—<\/strong> in the yaml file.<\/p>\n\n\n\nSo you can create your own OSD specification file and apply to create OSDs based on your specifications.<\/p>\n\n\n\n
In this guide, we will use the specification above to create and encrypt our OSDs.<\/p>\n\n\n\n
So, let’s create a YAML file to put out OSD drives specification parameters.<\/p>\n\n\n\n
cat > osd-spec.yaml << EOL\nservice_type: osd\nservice_id: cost_capacity\nservice_name: osd.cost_capacity\nplacement:\n host_pattern: '*'\nspec:\n data_devices:\n rotational: 1\n encrypted: true\n filter_logic: AND\n objectstore: bluestore\nEOL\n<\/code><\/pre>\n\n\n\nOnce you are ready to go, then use ceph command to apply the specifications into the cluster.<\/p>\n\n\n\n
Before you can write the changes, do a dry run;<\/p>\n\n\n\n
sudo ceph orch apply -i osd-spec.yaml --dry-run<\/code><\/pre>\n\n\n\nWhen you execute the command first time, you will have to wait a little bit, Preview data is being generated.. Please re-run this command in a bit.<\/strong><\/p>\n\n\n\nThen re-run the command;<\/p>\n\n\n\n
sudo ceph orch apply -i osd-spec.yaml --dry-run<\/code><\/pre>\n\n\n\nSample output;<\/p>\n\n\n\n
WARNING! Dry-Runs are snapshots of a certain point in time and are bound \nto the current inventory setup. If any of these conditions change, the \npreview will be invalid. Please make sure to have a minimal \ntimeframe between planning and applying the specs.\n####################\nSERVICESPEC PREVIEWS\n####################\n+---------+------+--------+-------------+\n|SERVICE |NAME |ADD_TO |REMOVE_FROM |\n+---------+------+--------+-------------+\n+---------+------+--------+-------------+\n################\nOSDSPEC PREVIEWS\n################\n+---------+---------------+--------+----------+----+-----+\n|SERVICE |NAME |HOST |DATA |DB |WAL |\n+---------+---------------+--------+----------+----+-----+\n|osd |cost_capacity |node04 |\/dev\/vdb |- |- |\n|osd |cost_capacity |node05 |\/dev\/vdb |- |- |\n+---------+---------------+--------+----------+----+-----+\n<\/code><\/pre>\n\n\n\nIf all is well, write the changes!<\/p>\n\n\n\n
sudo ceph orch apply -i osd-spec.yaml<\/code><\/pre>\n\n\n\nYou can also be able to apply the specifications from the standard input without using the file;<\/p>\n\n\n\n
cat << EOL | sudo ceph orch apply -i -\nservice_type: osd\nservice_id: cost_capacity\nservice_name: osd.cost_capacity\nplacement:\n host_pattern: '*'\nspec:\n data_devices:\n rotational: 1\n encrypted: true\n filter_logic: AND\n objectstore: bluestore\nEOL\n<\/code><\/pre>\n\n\n\nCheck the OSD status;<\/p>\n\n\n\n
sudo ceph osd status<\/code><\/pre>\n\n\n\nID HOST USED AVAIL WR OPS WR DATA RD OPS RD DATA STATE \n 0 node04 26.2M 99.9G 0 0 0 0 exists,up \n 1 node05 426M 99.5G 0 0 0 0 exists,up\n<\/code><\/pre>\n\n\n\nCheck Ceph status;<\/p>\n\n\n\n
sudo ceph -s<\/code><\/pre>\n\n\n\n cluster:\n id: ad7f576a-f1de-11ee-b470-fb0098ab30ad\n health: HEALTH_WARN\n OSD count 2 < osd_pool_default_size 3\n \n services:\n mon: 5 daemons, quorum node01,node02,node03,node05,node04 (age 5h)\n mgr: node02.ptcclf(active, since 5h), standbys: node04.oenghv\n osd: 2 osds: 2 up (since 58s), 2 in (since 87s)<\/strong>\n \n data:\n pools: 0 pools, 0 pgs\n objects: 0 objects, 0 B\n usage: 452 MiB used, 200 GiB \/ 200 GiB avail\n pgs:\n<\/code><\/pre>\n\n\n\nStatus from Dashboard;<\/p>\n\n\n\nFrom command line, you can add additional OSD nodes and label it.<\/p>\n\n\n\n
sudo ceph orch host add node06 192.168.122.202 osd03<\/code><\/pre>\n\n\n\nCeph will the scan the host for the availability of the un-allocated drives and set that up as OSD. It also enables encryption at the same time since you have already enabled OSD encryption, in cluster wide.<\/p>\n\n\n\n
sudo ceph -s<\/code><\/pre>\n\n\n\n cluster:\n id: ad7f576a-f1de-11ee-b470-fb0098ab30ad\n health: HEALTH_OK\n \n services:\n mon: 5 daemons, quorum node01,node02,node03,node05,node04 (age 6h)\n mgr: node02.ptcclf(active, since 6h), standbys: node04.oenghv\n osd: 3 osds: 2 up (since 20m), 3 in (since 10s)\n \n data:\n pools: 0 pools, 0 pgs\n objects: 0 objects, 0 B\n usage: 452 MiB used, 200 GiB \/ 200 GiB avail\n pgs:\n<\/code><\/pre>\n\n\n\nWe have added three OSD nodes;<\/p>\n\n\n\n
sudo ceph orch host ls<\/code><\/pre>\n\n\n\nHOST ADDR LABELS STATUS \nnode01 192.168.122.97 _admin,mon01,mgr01 \nnode02 192.168.122.98 mon02,mgr02 \nnode03 192.168.122.99 mon03 \nnode04 192.168.122.200 osd01 \nnode05 192.168.122.201 osd02 \nnode06 192.168.122.202 osd03 \n6 hosts in cluster\n<\/code><\/pre>\n\n\n\nEncrypt OSDs from the Command Line using ceph-volume Command<\/h4>\n\n\n\n Ceph provides yet another command that can be used to manually manage various OSD operations such as preparing, activating, creating, deleting, scanning OSD drives on the OSD nodes. ceph-volume<\/strong> can be used to enable encryption for the underlying OSD devices when they are being added into the Ceph storage cluster. Read more on the ceph-volume man page\/documentation<\/a>.<\/p>\n\n\n\nThe command is not usually installed by default. Therefore, on all OSD nodes, install the ceph-volume<\/code><\/strong> package to get the ceph-volume command.<\/p>\n\n\n\n(We are using Ubuntu OS on our OSD nodes, refer to documentation on how to install ceph-volume for other Linux distros<\/em>)<\/p>\n\n\n\nsudo su -<\/code><\/pre>\n\n\n\nwget -q -O- 'https:\/\/download.ceph.com\/keys\/release.asc' | \\\ngpg --dearmor -o \/etc\/apt\/trusted.gpg.d\/cephadm.gpg<\/code><\/pre>\n\n\n\necho deb https:\/\/download.ceph.com\/debian-reef\/ $(lsb_release -sc) main \\\n> \/etc\/apt\/sources.list.d\/cephadm.list<\/code><\/pre>\n\n\n\napt update<\/code><\/pre>\n\n\n\napt install ceph-volume<\/code><\/pre>\n\n\n\nYou can even check the help page;<\/p>\n\n\n\n
ceph-volume -h<\/code><\/pre>\n\n\n\nOnce the package is in place:<\/p>\n\n\n\n
\nproceed to define how the OSD client nodes will connect to the Ceph cluster monitor nodes. This can be achieved by editing or creating (if it does not exist) the ceph.conf<\/strong> file on the OSD client and defining the monitor nodes addresses.<\/li>\n\n\n\nSimilarly, in order for the ceph-volume client to bootstrap the OSD on the Ceph cluster, it requires the client OSD boostrap keys installed on the OSD nodes. The key is used to authenticate and authorize the OSD ceph client to boostrap OSD. You can place the key in the ceph.conf file in the client node.<\/li>\n<\/ul>\n\n\n\nSample config;<\/p>\n\n\n\n
sudo cat \/etc\/ceph\/ceph.conf<\/code><\/pre>\n\n\n\n# minimal ceph.conf for 8a7f658e-f3f5-11ee-9a19-4d1575fdfd98\n[global]\n\tfsid = 8a7f658e-f3f5-11ee-9a19-4d1575fdfd98\n\tmon_host = [v2:192.168.122.78:3300\/0,v1:192.168.122.78:6789\/0] [v2:192.168.122.79:3300\/0,v1:192.168.122.79:6789\/0] [v2:192.168.122.80:3300\/0,v1:192.168.122.80:6789\/0] [v2:192.168.122.90:3300\/0,v1:192.168.122.90:6789\/0] [v2:192.168.122.91:3300\/0,v1:192.168.122.91:6789\/0]\n<\/code><\/pre>\n\n\n\nTherefore, run this command on the admin node to copy the ceph.conf into the OSD nodes.<\/p>\n\n\n\n
ceph config generate-minimal-conf | ssh root@node03 '[ ! -d \/etc\/ceph ] && mkdir -p \/etc\/ceph; cat > \/etc\/ceph\/ceph.conf'<\/code><\/pre>\n\n\n\nThis installs ceph.conf on node03, which is one of my OSD nodes. Do the same on other OSD nodes.<\/p>\n\n\n\n
Append the Ceph client OSD bootstrap authentication key, client.bootstrap-osd<\/strong>, to the ceph.conf file copied above.<\/p>\n\n\n\nYou can confirm presence of this key on the Ceph admin node;<\/p>\n\n\n\n
sudo ceph auth list<\/code><\/pre>\n\n\n\nYou can check the key details by running;<\/p>\n\n\n\n
sudo ceph auth get client.bootstrap-osd<\/code><\/pre>\n\n\n\n[client.bootstrap-osd]\n\tkey = AQCWERFmBvnRFxAANvFxM+D\/QKGZwi40R91uWQ==\n\tcaps mon = \"allow profile bootstrap-osd\"\n<\/code><\/pre>\n\n\n\nSo, copy the bootstrap key and append it to ceph.conf file on the OSD client node.<\/p>\n\n\n\n
ceph auth get client.bootstrap-osd | ssh root@node03 '[ ! -d \/etc\/ceph ] && mkdir -p \/etc\/ceph; cat >> \/etc\/ceph\/ceph.conf'<\/code><\/pre>\n\n\n\nThis is how the OSD client Ceph configuration now looks like.<\/p>\n\n\n\n
root@node03:~# cat \/etc\/ceph\/ceph.conf<\/code><\/pre>\n\n\n\n# minimal ceph.conf for 8a7f658e-f3f5-11ee-9a19-4d1575fdfd98\n[global]\n\tfsid = 8a7f658e-f3f5-11ee-9a19-4d1575fdfd98\n\tmon_host = [v2:192.168.122.78:3300\/0,v1:192.168.122.78:6789\/0] [v2:192.168.122.79:3300\/0,v1:192.168.122.79:6789\/0] [v2:192.168.122.90:3300\/0,v1:192.168.122.90:6789\/0] [v2:192.168.122.91:3300\/0,v1:192.168.122.91:6789\/0]\n[client.bootstrap-osd]\n\tkey = AQCWERFmBvnRFxAANvFxM+D\/QKGZwi40R91uWQ==\n\tcaps mon = \"allow profile bootstrap-osd\"\n<\/code><\/pre>\n\n\n\nNow that the Ceph OSD client authentication to the Ceph cluster is sorted, proceed to create and OSD and enable LUKs encryption using ceph-volume command.<\/p>\n\n\n\n
lsblk<\/code><\/pre>\n\n\n\nNAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS\nvda 252:0 0 50G 0 disk \n\u251c\u2500vda1 252:1 0 1M 0 part \n\u251c\u2500vda2 252:2 0 2G 0 part \/boot\n\u2514\u2500vda3 252:3 0 48G 0 part \n \u2514\u2500ubuntu--vg-ubuntu--lv 253:0 0 48G 0 lvm \/\nvdb 252:16 0 100G 0 disk \n<\/code><\/pre>\n\n\n\nceph-volume lvm create --bluestore --data \/dev\/vdb --dmcrypt<\/code><\/pre>\n\n\n\nIf the command proceeded successfully, you should see;<\/p>\n\n\n\n
...\nRunning command: \/usr\/bin\/systemctl enable --runtime ceph-osd@0\nRunning command: \/usr\/bin\/systemctl start ceph-osd@0\n--> ceph-volume lvm activate successful for osd ID: 0\n--> ceph-volume lvm create successful for: \/dev\/vdb\n<\/code><\/pre>\n\n\n\n\nNote that the Ceph status may show OSDs added via ceph-volume as stray. This is because in a cephadm managed cluster, then the cluster expects the OSDs to be added via cephadm command.<\/p>\n\n\n\n
This should confirm that the OSD is fine and works as expected despite the stray alert.<\/p>\n\n\n\n
ceph osd status<\/code><\/pre>\n\n\n\nID HOST USED AVAIL WR OPS WR DATA RD OPS RD DATA STATE \n 0 node03 27.3M 99.9G 0 0 0 0 exists,up \n<\/code><\/pre>\n<\/blockquote>\n\n\n\nVerifying Ceph OSD Encryption<\/h3>\n\n\n\n Will that said and done, how can you actually verify and confirm that your OSDs are indeed encrypted?<\/p>\n\n\n\n
Use blkid<\/strong> or lsblk<\/strong> commands<\/h4>\n\n\n\nAs you are already aware, Ceph uses LUKS<\/strong> to encrypt devices. LUKS utilizes dm-crypt<\/strong> (device mapper crypt) to perform the actual encryption and decryption operations.<\/p>\n\n\n\nTherefore, to check if your OSD drives have been encrypted, you can use lsblk<\/strong> or blkid<\/strong> commands to check if the type<\/strong> device has been set to crypt<\/strong>.<\/p>\n\n\n\nlsblk<\/code><\/pre>\n\n\n\n