{"id":22174,"date":"2024-04-20T19:26:41","date_gmt":"2024-04-20T16:26:41","guid":{"rendered":"https:\/\/kifarunix.com\/?p=22174"},"modified":"2024-04-20T19:26:45","modified_gmt":"2024-04-20T16:26:45","slug":"rescue-openstack-instance-using-systemrescue-image","status":"publish","type":"post","link":"https:\/\/kifarunix.com\/rescue-openstack-instance-using-systemrescue-image\/","title":{"rendered":"Rescue OpenStack Instance using SystemRescue Image"},"content":{"rendered":"\n<p>In this tutorial, you will learn how to rescue OpenStack instance using <a href=\"https:\/\/www.system-rescue.org\/\" target=\"_blank\" rel=\"noopener\">SystemRescue<\/a> image. According to <a href=\"https:\/\/www.system-rescue.org\/\" target=\"_blank\" rel=\"noreferrer noopener\">SystemRescure Homepage<\/a>;<\/p>\n\n\n\n<blockquote class=\"wp-block-quote has-small-font-size is-layout-flow wp-block-quote-is-layout-flow\">\n<p>SystemRescue (formerly known as SystemRescueCd) is a Linux system rescue toolkit available as a bootable medium for administrating or repairing your system and data after a crash. It aims to provide an easy way to carry out admin tasks on your computer, such as creating and editing the hard disk partitions. It comes with a lot of\u00a0Linux system utilities\u00a0such as GParted, fsarchiver, filesystem tools and basic tools (editors, midnight commander, network tools). It can be used for both Linux and\u00a0windows\u00a0computers, and on desktops as well as servers.\u00a0<\/p>\n<\/blockquote>\n\n\n\n<div class=\"wp-block-rank-math-toc-block\" id=\"rank-math-toc\"><h2>Table of Contents<\/h2><nav><ul><li><a href=\"#rescue-open-stack-instance-using-system-rescue-image\">Rescue OpenStack Instance using SystemRescue Image<\/a><ul><li><a href=\"#common-scenarios-to-use-system-rescure-on-open-stack\">Common Scenarios to use SystemRescure on OpenStack<\/a><\/li><li><a href=\"#precautions-to-take-before-putting-instance-in-rescue-mode\">Precautions to take Before Putting Instance in Rescue Mode<\/a><\/li><li><a href=\"#download-and-import-system-rescue-image-into-openstack\">Download and Import SystemRescue Image into Openstack<\/a><\/li><li><a href=\"#rescueing-open-stack-instance\">Rescueing OpenStack Instance<\/a><ul><li><a href=\"#rescueing-instance-from-command-line\">Rescueing Instance from command line<\/a><\/li><li><a href=\"#rescue-instance-on-open-stack-horizon\">Rescue Instance on OpenStack Horizon<\/a><\/li><\/ul><\/li><li><a href=\"#access-and-troubleshoot-instance-from-rescue-state\">Access and Troubleshoot Instance from Rescue State<\/a><ul><li><a href=\"#accessing-rescued-instance-on-command-line\">Accessing Rescued Instance on Command Line<\/a><\/li><li><a href=\"#accessing-rescued-instance-from-horizon\">Accessing Rescued Instance from Horizon<\/a><\/li><\/ul><\/li><li><a href=\"#troubleshooting-open-stack-instance-in-rescue-mode\">Troubleshooting OpenStack Instance in Rescue Mode<\/a><ul><li><a href=\"#mounting-instance-filesystem\">Mounting Instance Filesystem<\/a><\/li><li><a href=\"#troubleshooting-instance-from-system-rescue\">Troubleshooting Instance from SystemRescue<\/a><\/li><\/ul><\/li><li><a href=\"#unrescue-the-instance\">Unrescue the Instance<\/a><\/li><\/ul><\/li><\/ul><\/nav><\/div>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"rescue-open-stack-instance-using-system-rescue-image\">Rescue OpenStack Instance using SystemRescue Image<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"common-scenarios-to-use-system-rescure-on-open-stack\">Common Scenarios to use SystemRescure on OpenStack<\/h3>\n\n\n\n<p>So, what are the common situations that might prompt the need to use SystemRescue to rescue an OpenStack instance? Some of the situations may include;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Filesystem Corruption<\/strong>: If an OpenStack instance experienced issues with filesystem corruption that prevents it from booting correctly, then you might need to boot if from SystemRescue to be able to login and repair the filesystem errors.<\/li>\n\n\n\n<li><strong>Lost Passwords<\/strong>: While you can boot you instance into a single user mode and reset your passwords, sometime it is challenging to get access to instance grub menu. As such, SystemRescure can be used instead.<\/li>\n\n\n\n<li><strong>Kernel Panic<\/strong>: If you instance crashes leading to Kernel panic, that renders your machine inaccessible, a quicker way to fix it would be to boot it using SystemRescue to diagnose and fix any issue.<\/li>\n\n\n\n<li><strong><strong>Boot Loader Issues<\/strong><\/strong>: You can also fix your instance boot issues using SystemRescue image.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"precautions-to-take-before-putting-instance-in-rescue-mode\">Precautions to take Before Putting Instance in Rescue Mode<\/h3>\n\n\n\n<p>Remember when you put your instance rescue mode, that instance wont be accessible until you have unrescue it. As such, here are some of the precautions that you might need to put in place.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Backup<\/strong> your data: If there is any critical data stored on the instance, be sure to have backed it up and validated the backup before taking the instance into the rescue mode.<\/li>\n\n\n\n<li><strong>Plan Downtime<\/strong>: If the instance is actively being used, you can plan downtime so you can fix any would be issue during a maintenance window period or off-peak hours to minimize service interruption.<\/li>\n\n\n\n<li><strong>Implement Redundancy<\/strong>: If your instance is running a critical service, you can consider setting it up in high availability to ensure redundancy and continued service availability when you put your other node in rescue mode.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"download-and-import-system-rescue-image-into-openstack\">Download and Import SystemRescue Image into Openstack<\/h3>\n\n\n\n<p>To be able to use SystemRescue image on OpenStack to rescue your instances, you have to first import it into OpenStack.<\/p>\n\n\n\n<p>Therefore, <a href=\"https:\/\/www.system-rescue.org\/Download\/\" target=\"_blank\" rel=\"noopener\">download<\/a> the image into your controller node;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>wget https:\/\/fastly-cdn.system-rescue.org\/releases\/11.00\/systemrescue-11.00-amd64.iso<\/code><\/pre>\n\n\n\n<p>Similarly, download the checksum hash file to verify integrity of the image.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>wget https:\/\/www.system-rescue.org\/releases\/11.00\/systemrescue-11.00-amd64.iso.sha512<\/code><\/pre>\n\n\n\n<p>Next, verify integrity of the image. Ensure both the ISO file and Checksum file are in the same working directory.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sha512sum --check systemrescue-11.00-amd64.iso.sha512<\/code><\/pre>\n\n\n\n<p>If all good, you should get such an output;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>systemrescue-11.00-amd64.iso: OK<\/code><\/pre>\n\n\n\n<p>You can now import image into OpenStack. To import image into OpenStack, you can use use <strong>glance<\/strong> or <strong>openstack<\/strong> command.<\/p>\n\n\n\n<p>Activate your OpenStack environment and load the credentials<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>source $HOME\/kolla-ansible\/bin\/activate<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>source \/etc\/kolla\/admin-openrc.sh<\/code><\/pre>\n\n\n\n<p>Then create the SystemRescue image;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>glance image-create --name systemrescue \\\n\t--file .\/systemrescue-11.00-amd64.iso \\\n\t--disk-format iso \\\n\t--container-format bare \\\n\t--progress\n<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>+------------------+----------------------------------------------------------------------------------+\n| Property         | Value                                                                            |\n+------------------+----------------------------------------------------------------------------------+\n| checksum         | fee7f202ba632552dbaf82a89c2438af                                                 |\n| container_format | bare                                                                             |\n| created_at       | 2024-04-20T07:25:20Z                                                             |\n| direct_url       | rbd:\/\/17ef548c-f68b-11ee-9a19-4d1575fdfd98\/glance-                               |\n|                  | images\/1c0c448c-e766-4b8a-ac01-4b553c888074\/snap                                 |\n| disk_format      | iso                                                                              |\n| id               | 1c0c448c-e766-4b8a-ac01-4b553c888074                                             |\n| min_disk         | 0                                                                                |\n| min_ram          | 0                                                                                |\n| name             | systemrescue                                                                     |\n| os_hash_algo     | sha512                                                                           |\n| os_hash_value    | 10a3145a5101c00977091e7c946086edcc5e4acf8dfb57b1e912fdf27c23899cedc12cf14ef503a1 |\n|                  | 8a1018f917066149957d7460f3c080e2891af64cac05bc35                                 |\n| os_hidden        | False                                                                            |\n| owner            | dee76bfb1767468ba250225944203193                                                 |\n| protected        | False                                                                            |\n| size             | 894435328                                                                        |\n| status           | active                                                                           |\n| stores           | rbd                                                                              |\n| tags             | []                                                                               |\n| updated_at       | 2024-04-20T07:25:30Z                                                             |\n| virtual_size     | Not available                                                                    |\n| visibility       | shared                                                                           |\n+------------------+----------------------------------------------------------------------------------+\n<\/code><\/pre>\n\n\n\n<p>If you want to use <strong>openstack<\/strong> command;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>openstack image create systemrescuecd \\\n        --file .\/systemrescue-11.00-amd64.iso \\\n        --disk-format iso \\\n        --container-format bare \\\n        --progress\n<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>+------------------+----------------------------------------------------------------------------------------------------------------------------------------------------+\n| Field            | Value                                                                                                                                              |\n+------------------+----------------------------------------------------------------------------------------------------------------------------------------------------+\n| container_format | bare                                                                                                                                               |\n| created_at       | 2024-04-20T07:27:08Z                                                                                                                               |\n| disk_format      | iso                                                                                                                                                |\n| file             | \/v2\/images\/83b6b25a-717d-4263-ad45-150cd301eb30\/file                                                                                               |\n| id               | 83b6b25a-717d-4263-ad45-150cd301eb30                                                                                                               |\n| min_disk         | 0                                                                                                                                                  |\n| min_ram          | 0                                                                                                                                                  |\n| name             | systemrescuecd                                                                                                                                     |\n| owner            | dee76bfb1767468ba250225944203193                                                                                                                   |\n| properties       | os_hidden='False', owner_specified.openstack.md5='', owner_specified.openstack.object='images\/systemrescuecd', owner_specified.openstack.sha256='' |\n| protected        | False                                                                                                                                              |\n| schema           | \/v2\/schemas\/image                                                                                                                                  |\n| status           | queued                                                                                                                                             |\n| tags             |                                                                                                                                                    |\n| updated_at       | 2024-04-20T07:27:08Z                                                                                                                               |\n| visibility       | shared                                                                                                                                             |\n+------------------+----------------------------------------------------------------------------------------------------------------------------------------------------+\n<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"rescueing-open-stack-instance\">Rescueing OpenStack Instance<\/h3>\n\n\n\n<p>As already mentioned, there are various situations that might prompt you to consider using SystemRescue to rescue your OpenStack instance.<\/p>\n\n\n\n<p>In my case, I had forgotten the instance credentials and I couldn&#8217;t easily access Grub menu from the OpenStack console. The only chance I had was with using SystemRescue.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>openstack server list<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>+--------------------------------------+----------+--------+------------------+--------+--------+\n| ID                                   | Name     | Status | Networks         | Image  | Flavor |\n+--------------------------------------+----------+--------+------------------+--------+--------+\n| c6efdc70-ed41-4d34-9317-c3eb878df62b | jammy    | ACTIVE | net=10.10.10.156 | jammy  | minif  |\n| 4e5b93a9-9de9-42e4-9419-5df8a3bd07cc | cephtest | ACTIVE | net=10.10.10.142 | cirros | mini   |\n+--------------------------------------+----------+--------+------------------+--------+--------+\n<\/code><\/pre>\n\n\n\n<p>Or<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1617\" height=\"581\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/04\/openstack-instances.png?v=1713598982\" alt=\"\" class=\"wp-image-22192\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/04\/openstack-instances.png?v=1713598982 1617w, https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/04\/openstack-instances-768x276.png?v=1713598982 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/04\/openstack-instances-1536x552.png?v=1713598982 1536w\" sizes=\"(max-width: 1617px) 100vw, 1617px\" \/><\/figure>\n\n\n\n<p>That is the instance that I will be working on!<\/p>\n\n\n\n<p>There are two ways in which you can rescue you OpenStack instance.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Rescueing Instance from command line<\/li>\n\n\n\n<li>Rescueing Instance from Horizon Dashboard<\/li>\n<\/ul>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"rescueing-instance-from-command-line\">Rescueing Instance from command line<\/h4>\n\n\n\n<p>To rescue an instance from command line, you can use the command;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>openstack server rescue &#91;--image &lt;image>] &#91;--password &lt;password>] &lt;server><\/code><\/pre>\n\n\n\n<p>you need to know the instance\/server name or ID, as well the SystemRescue image name or ID.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>openstack server list<\/code><\/pre>\n\n\n\n<p>See our output above.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>openstack image list<\/code><\/pre>\n\n\n\n<p>Sample output;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>+--------------------------------------+--------------+--------+\n| ID                                   | Name         | Status |\n+--------------------------------------+--------------+--------+\n| 024bd4c2-8ae6-4020-9f61-fb463c4b0b16 | cirros       | active |\n| 097705d6-1d51-4c97-9c4a-12c7514ec3a7 | jammy        | active |\n| 1c0c448c-e766-4b8a-ac01-4b553c888074 | systemrescue | active |\n+--------------------------------------+--------------+--------+\n<\/code><\/pre>\n\n\n\n<p>Now that we have the server name\/ID as well the rescue image name\/id, proceed to stop and rescue the instance.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>openstack server stop c6efdc70-ed41-4d34-9317-c3eb878df62b<\/code><\/pre>\n\n\n\n<p>Confirm that the instance that instance is shutoff;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>openstack server list<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>+--------------------------------------+----------+---------+------------------+--------+--------+\n| ID                                   | Name     | Status  | Networks         | Image  | Flavor |\n+--------------------------------------+----------+---------+------------------+--------+--------+\n| c6efdc70-ed41-4d34-9317-c3eb878df62b | jammy    | SHUTOFF | net=10.10.10.156 | jammy  | minif  |\n| 4e5b93a9-9de9-42e4-9419-5df8a3bd07cc | cephtest | ACTIVE  | net=10.10.10.142 | cirros | mini   |\n+--------------------------------------+----------+---------+------------------+--------+--------+\n<\/code><\/pre>\n\n\n\n<p>Now, rescue the instance;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>openstack server rescue --image 1c0c448c-e766-4b8a-ac01-4b553c888074 <strong>c6efdc70-ed41-4d34-9317-c3eb878df62b<\/strong><\/code><\/pre>\n\n\n\n<p>This will start the instance and boot into SystemRescue.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"rescue-instance-on-open-stack-horizon\">Rescue Instance on OpenStack Horizon<\/h4>\n\n\n\n<p>You can as well rescue instance from OpenStack horizon;<\/p>\n\n\n\n<p>Assuming you have already imported a SystemRescue image into OpenStack;<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1617\" height=\"557\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/04\/system-rescue-image.png?v=1713619344\" alt=\"systemrescue image on openstack\" class=\"wp-image-22196\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/04\/system-rescue-image.png?v=1713619344 1617w, https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/04\/system-rescue-image-768x265.png?v=1713619344 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/04\/system-rescue-image-1536x529.png?v=1713619344 1536w\" sizes=\"(max-width: 1617px) 100vw, 1617px\" \/><\/figure>\n\n\n\n<p>Then select your project and navigate to <strong>Project > Compute > Instances<\/strong>.<\/p>\n\n\n\n<p>Identify the instance you want to rescue under the actions drop down button on the right of the instance, click <strong>Rescue Instance<\/strong>.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1619\" height=\"746\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/04\/allow-unattended-remove-access.png?v=1713619369\" alt=\"\" class=\"wp-image-22197\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/04\/allow-unattended-remove-access.png?v=1713619369 1619w, https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/04\/allow-unattended-remove-access-768x354.png?v=1713619369 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/04\/allow-unattended-remove-access-1536x708.png?v=1713619369 1536w\" sizes=\"(max-width: 1619px) 100vw, 1619px\" \/><\/figure>\n\n\n\n<p>Next, select a rescue image (we use SystemRescue image here) and optionally set a password on the rescued instance.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1605\" height=\"524\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/04\/select-resue-image.png?v=1713619598\" alt=\"\" class=\"wp-image-22198\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/04\/select-resue-image.png?v=1713619598 1605w, https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/04\/select-resue-image-768x251.png?v=1713619598 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/04\/select-resue-image-1536x501.png?v=1713619598 1536w\" sizes=\"(max-width: 1605px) 100vw, 1605px\" \/><\/figure>\n\n\n\n<p>Click <strong>Confirm<\/strong>, to shutdown the instance and boot it from the rescue image.<\/p>\n\n\n\n<p>The state of the image now should show as <strong>Rescue<\/strong>.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1617\" height=\"479\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/04\/instance-rescued.png?v=1713619777\" alt=\"\" class=\"wp-image-22199\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/04\/instance-rescued.png?v=1713619777 1617w, https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/04\/instance-rescued-768x228.png?v=1713619777 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/04\/instance-rescued-1536x455.png?v=1713619777 1536w\" sizes=\"(max-width: 1617px) 100vw, 1617px\" \/><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"access-and-troubleshoot-instance-from-rescue-state\">Access and Troubleshoot Instance from Rescue State<\/h3>\n\n\n\n<p>You can access a rescued instance from command line or from horizon.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"accessing-rescued-instance-on-command-line\">Accessing Rescued Instance on Command Line<\/h4>\n\n\n\n<p>To access a rescued instance from the terminal, you need to identify the compute node on which the instance is hosted, as well the <strong>instance_name<\/strong>. You can get these details using <strong>openstack server show<\/strong> command or from Horizon.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>openstack server show c6efdc70-ed41-4d34-9317-c3eb878df62b<\/code><\/pre>\n\n\n\n<p>Sample details;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>+-------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+\n| Field                               | Value                                                                                                                                     |\n+-------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+\n| OS-DCF:diskConfig                   | AUTO                                                                                                                                      |\n| OS-EXT-AZ:availability_zone         | nova                                                                                                                                      |\n| OS-EXT-SRV-ATTR:host                | openstack                                                                                                                                 |\n| OS-EXT-SRV-ATTR:hostname            | jammy                                                                                                                                     |\n| OS-EXT-SRV-ATTR:hypervisor_hostname | openstack                                                                                                                                 |\n| OS-EXT-SRV-ATTR:instance_name       | instance-0000000b                                                                                                                         |\n| OS-EXT-SRV-ATTR:kernel_id           |                                                                                                                                           |\n| OS-EXT-SRV-ATTR:launch_index        | 0                                                                                                                                         |\n| OS-EXT-SRV-ATTR:ramdisk_id          |                                                                                                                                           |\n| OS-EXT-SRV-ATTR:reservation_id      | r-mvg279bn                                                                                                                                |\n| OS-EXT-SRV-ATTR:root_device_name    | \/dev\/vda                                                                                                                                  |\n| OS-EXT-SRV-ATTR:user_data           | None                                                                                                                                      |\n| OS-EXT-STS:power_state              | Running                                                                                                                                   |\n| OS-EXT-STS:task_state               | None                                                                                                                                      |\n| OS-EXT-STS:vm_state                 | rescued                                                                                                                                   |\n| OS-SRV-USG:launched_at              | 2024-04-20T13:27:51.000000                                                                                                                |\n| OS-SRV-USG:terminated_at            | None                                                                                                                                      |\n| accessIPv4                          |                                                                                                                                           |\n| accessIPv6                          |                                                                                                                                           |\n| addresses                           | net=10.10.10.156                                                                                                                          |\n| config_drive                        |                                                                                                                                           |\n| created                             | 2024-04-16T13:39:31Z                                                                                                                      |\n| description                         | None                                                                                                                                      |\n| flavor                              | description=, disk='5', ephemeral='0', , id='minif', is_disabled=, is_public='True', location=, name='minif', original_name='minif',      |\n|                                     | ram='2048', rxtx_factor=, swap='0', vcpus='1'                                                                                             |\n| hostId                              | f32cbc449e29ca574b72dd876976e8c4cf44d4c86b5b39617f322080                                                                                  |\n| host_status                         | UP                                                                                                                                        |\n| id                                  | c6efdc70-ed41-4d34-9317-c3eb878df62b                                                                                                      |\n| image                               | jammy (097705d6-1d51-4c97-9c4a-12c7514ec3a7)                                                                                              |\n| key_name                            | None                                                                                                                                      |\n| locked                              | False                                                                                                                                     |\n| locked_reason                       | None                                                                                                                                      |\n| name                                | jammy                                                                                                                                     |\n| progress                            | None                                                                                                                                      |\n| project_id                          | dee76bfb1767468ba250225944203193                                                                                                          |\n| properties                          |                                                                                                                                           |\n| security_groups                     | name='default'                                                                                                                            |\n| server_groups                       | []                                                                                                                                        |\n| status                              | RESCUE                                                                                                                                    |\n| tags                                |                                                                                                                                           |\n| trusted_image_certificates          | None                                                                                                                                      |\n| updated                             | 2024-04-20T13:27:51Z                                                                                                                      |\n| user_id                             | 885eb7e423154ed781cdd0a71cb5221f                                                                                                          |\n| volumes_attached                    |                                                                                                                                           |\n+-------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+\n<\/code><\/pre>\n\n\n\n<p>The details i need are:<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>| OS-EXT-SRV-ATTR:hypervisor_hostname | openstack                                                 \n| OS-EXT-SRV-ATTR:instance_name       | instance-0000000b \n<\/code><\/pre>\n\n\n\n<p>So, login to the node, in this case, openstack node;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ssh username@compute-node<\/code><\/pre>\n\n\n\n<p>Replace the details accordingly.<\/p>\n\n\n\n<p>Then use <strong>virsh<\/strong> command to login to instance console. We are using Kolla-ansible deployed OpenStack hence, our OpenStack services are running as Docker containers. We will therefore connect to Nova Libvirt service to get to login to the instance.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>docker exec -it nova_libvirt bash<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>virsh list<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code> Id   Name                State\n-----------------------------------\n 23   instance-00000008   running\n 25   instance-0000000b   running\n<\/code><\/pre>\n\n\n\n<p>Then;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>virsh console instance-0000000b<\/code><\/pre>\n\n\n\n<p>Or simply;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>docker exec -it nova_libvirt virsh list<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>docker exec -it nova_libvirt virsh console instance-0000000b<\/code><\/pre>\n\n\n\n<p>You are given such prompt;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>Connected to domain 'instance-0000000b'\nEscape character is ^] (Ctrl + ])\n<\/code><\/pre>\n\n\n\n<p>Press the Escape character combination to jump into shell. If it doesn&#8217;t work, try <strong>CTLR+C<\/strong>.<\/p>\n\n\n\n<p>In my case, I was not able to get to the shell prompt. Thus, we will resort to using Horizon console!<\/p>\n\n\n\n<p>From the console, you can do any administrative tasks to fix the issue you intended to fix.<\/p>\n\n\n\n<p>To exit the <strong>virsh console<\/strong>, use <strong>CTRL+Shift+]<\/strong>.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"accessing-rescued-instance-from-horizon\">Accessing Rescued Instance from Horizon<\/h4>\n\n\n\n<p>Navigate to your project instances page on Horizon and click the instance that you had rescued to open it, and click <strong>console<\/strong> to access the console.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1614\" height=\"753\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/04\/openstack-instance-horizon-console.png?v=1713624525\" alt=\"\" class=\"wp-image-22202\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/04\/openstack-instance-horizon-console.png?v=1713624525 1614w, https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/04\/openstack-instance-horizon-console-768x358.png?v=1713624525 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/04\/openstack-instance-horizon-console-1536x717.png?v=1713624525 1536w\" sizes=\"(max-width: 1614px) 100vw, 1614px\" \/><\/figure>\n\n\n\n<p>To open the console only on new browser tab, right click the link, &#8220;<strong>click here to show only console<\/strong>&#8221; and open in new tab.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"troubleshooting-open-stack-instance-in-rescue-mode\">Troubleshooting OpenStack Instance in Rescue Mode<\/h3>\n\n\n\n<p>From the console of the instance, either on the command line or on horizon, you can now do your administrative tasks or troubleshooting.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"mounting-instance-filesystem\">Mounting Instance Filesystem<\/h4>\n\n\n\n<p>If you need to update your instance configuration for example, then you have to mount your root filesystem.<\/p>\n\n\n\n<p>Similarly, if you have other drive partitions for boot\/esp and you need to make changes related to processes whose configurations resides on such partitions, you have to mount it as well.<\/p>\n\n\n\n<p>SystemRescue has almost all Linux usual commands! Let&#8217;s check the drives<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>lsblk<\/code><\/pre>\n\n\n\n<p>From the screenshot, vdb is our instance drive, where vdb1 is the rootFS, vdb15 are ESP Boot partitions.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1470\" height=\"526\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/04\/list-block-drives.png?v=1713625401\" alt=\"\" class=\"wp-image-22203\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/04\/list-block-drives.png?v=1713625401 1470w, https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/04\/list-block-drives-768x275.png?v=1713625401 768w\" sizes=\"(max-width: 1470px) 100vw, 1470px\" \/><\/figure>\n\n\n\n<p>So, you can mount the drives;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>mkdir \/mnt\/vdb<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>mount \/dev\/vdb1 \/mnt\/vdb\/<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>mount \/dev\/vdb15 \/mnt\/vdb\/boot\/efi<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"troubleshooting-instance-from-system-rescue\">Troubleshooting Instance from SystemRescue<\/h4>\n\n\n\n<p>From here, you can edit some files directly, for example, if you want to set static IP address;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>vim \/mnt\/vdb\/etc\/netplan\/50-cloud-init.yaml<\/code><\/pre>\n\n\n\n<p>Make your appropriate changes and save the configurations.<\/p>\n\n\n\n<p>You can also <strong>chroot<\/strong> into your instance root directory and perform your troubleshooting.<\/p>\n\n\n\n<blockquote class=\"wp-block-quote has-small-font-size is-layout-flow wp-block-quote-is-layout-flow\">\n<p>A <strong><em>chroot<\/em><\/strong> is an operation that changes the apparent root directory for the current running process and their children. A program that is run in such a modified environment cannot access files and commands outside that environmental directory tree. This modified environment is called a chroot jail.<\/p>\n\n\n\n<p>Reasoning<br>Changing root is commonly done for performing system maintenance on systems where booting and\/or logging in is no longer possible. Common examples are:<\/p>\n\n\n\n<p>Reinstalling the boot loader.<br>Rebuilding the initramfs image.<br>Upgrading or downgrading packages.<br>Resetting a forgotten password.<br>Building packages in a clean chroot.<\/p>\n<cite><a href=\"https:\/\/wiki.archlinux.org\/title\/chroot\" target=\"_blank\" rel=\"noreferrer noopener\">ArchWiki<\/a><\/cite><\/blockquote>\n\n\n\n<p>To chroot into your instance rootfs in SystemRescue;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>chroot \/mnt\/vdb<\/code><\/pre>\n\n\n\n<p>From chroot environment, you can perform any action you need to.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"unrescue-the-instance\">Unrescue the Instance<\/h3>\n\n\n\n<p>Once you are done fixing or troubleshooting your instance, you can exit the rescue system and unrescue the instance.<\/p>\n\n\n\n<p>If you are doing from command line, exit the console and;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>openstack server unrescue &lt;INSTANCE_ID><\/code><\/pre>\n\n\n\n<p>E.g<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>openstack server unrescue  <strong>c6efdc70-ed41-4d34-9317-c3eb878df62b<\/strong><\/code><\/pre>\n\n\n\n<p>Or from the Horizon;<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1619\" height=\"593\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/04\/unrescue-instance-horizon.png?v=1713628718\" alt=\"\" class=\"wp-image-22209\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/04\/unrescue-instance-horizon.png?v=1713628718 1619w, https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/04\/unrescue-instance-horizon-768x281.png?v=1713628718 768w, https:\/\/kifarunix.com\/wp-content\/uploads\/2024\/04\/unrescue-instance-horizon-1536x563.png?v=1713628718 1536w\" sizes=\"(max-width: 1619px) 100vw, 1619px\" \/><\/figure>\n\n\n\n<p>And that is.<\/p>\n\n\n\n<p>Once your instance is up, verify if it is working as expected.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>In this tutorial, you will learn how to rescue OpenStack instance using SystemRescue image. According to SystemRescure Homepage; SystemRescue (formerly known as SystemRescueCd) is a<\/p>\n","protected":false},"author":10,"featured_media":22211,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"rank_math_lock_modified_date":false,"footnotes":""},"categories":[121,1885,1886],"tags":[7453,7451,7452],"class_list":["post-22174","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-howtos","category-cloud-compute","category-openstack","tag-openstack-rescue-instance","tag-rescue-openstack-instance","tag-systemrescude-image-openstack","generate-columns","tablet-grid-50","mobile-grid-100","grid-parent","grid-50","resize-featured-image"],"_links":{"self":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/22174"}],"collection":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/users\/10"}],"replies":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/comments?post=22174"}],"version-history":[{"count":18,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/22174\/revisions"}],"predecessor-version":[{"id":22210,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/22174\/revisions\/22210"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media\/22211"}],"wp:attachment":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media?parent=22174"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/categories?post=22174"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/tags?post=22174"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}