{"id":19138,"date":"2023-11-12T20:25:27","date_gmt":"2023-11-12T17:25:27","guid":{"rendered":"https:\/\/kifarunix.com\/?p=19138"},"modified":"2024-03-10T15:08:21","modified_gmt":"2024-03-10T12:08:21","slug":"add-controller-nodes-into-existing-openstack-cluster-using-kolla-ansible","status":"publish","type":"post","link":"https:\/\/kifarunix.com\/add-controller-nodes-into-existing-openstack-cluster-using-kolla-ansible\/","title":{"rendered":"Add Controller Nodes into Existing OpenStack Cluster using Kolla-Ansible"},"content":{"rendered":"\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1062\" height=\"593\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/11\/openstack-kolla-ansible-add-controllers.png?v=1699809603\" alt=\"Add Controller Nodes into Existing OpenStack Cluster using Kolla-Ansible\" class=\"wp-image-19214\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/11\/openstack-kolla-ansible-add-controllers.png?v=1699809603 1062w, https:\/\/kifarunix.com\/wp-content\/uploads\/2023\/11\/openstack-kolla-ansible-add-controllers-768x429.png?v=1699809603 768w\" sizes=\"(max-width: 1062px) 100vw, 1062px\" \/><\/figure>\n\n\n\n<p>Is it possible to add controller nodes into existing <a href=\"https:\/\/www.openstack.org\/\" target=\"_blank\" rel=\"noreferrer noopener\">OpenStack<\/a> cluster using <a href=\"https:\/\/docs.openstack.org\/kolla-ansible\/latest\/\" target=\"_blank\" rel=\"noreferrer noopener\">Kolla-Ansible<\/a>? Of course, yes! In this blog post, you will learn how to add controller nodes to existing OpenStack cluster to boost the performance and provide high availability. Kolla-Ansible is a powerful tool that streamlines the deployment and management of OpenStack services using Docker containers. If you have <a href=\"https:\/\/kifarunix.com\/?s=deploy+openstack+kolla+ansible\" target=\"_blank\" rel=\"noreferrer noopener\">deployed your OpenStack using Kolla-Ansible<\/a>, then this guide is for you!<\/p>\n\n\n\n<div class=\"wp-block-rank-math-toc-block\" id=\"rank-math-toc\"><h2>Table of Contents<\/h2><nav><ul><li><a href=\"#use-kolla-ansible-to-add-controller-nodes-into-existing-open-stack-cluster\">Use Kolla-Ansible to Add Controller Nodes into Existing OpenStack Cluster<\/a><ul><li><a href=\"#what-is-a-controller-node-and-why-is-it-important\">What is a Controller Node and Why is it Important?<\/a><\/li><li><a href=\"#what-is-the-recommended-number-of-controller-nodes-in-openstack\">What is the recommended number of Controller Nodes in Openstack?<\/a><\/li><li><a href=\"#ideal-timing-for-adding-controller-nodes-in-a-production-environment\">Ideal Timing for Adding Controller Nodes in a Production Environment:<\/a><\/li><li><a href=\"#prepare-the-nodes-for-addition-into-open-stack\">Prepare the Nodes for Addition into OpenStack<\/a><\/li><li><a href=\"#are-you-using-shared-storage-for-glance-images\">Are you using Shared Storage for Glance Images?<\/a><\/li><li><a href=\"#copy-deployment-user-ssh-keys-from-control-node-to-new-controller-node\">Copy Deployment User SSH Keys from Control Node to New Controller Node<\/a><\/li><li><a href=\"#update-kolla-ansible-inventory\">Update Kolla-Ansible Inventory<\/a><\/li><li><a href=\"#enable-ha-proxy-for-high-availability-and-load-balancing\">Enable HAProxy for High Availability and Load Balancing<\/a><\/li><li><a href=\"#activate-kolla-ansible-virtual-environment\">Activate Kolla-Ansible Virtual Environment<\/a><\/li><li><a href=\"#test-connectivity-to-the-node\">Test Connectivity to the Node<\/a><\/li><li><a href=\"#bootstrap-the-new-controller-nodes\">Bootstrap the new Controller Nodes<\/a><\/li><li><a href=\"#run-pre-deployment-checks-on-the-new-controller-nodes\">Run Pre-Deployment Checks on the New Controller Nodes<\/a><\/li><li><a href=\"#deploy-required-services-docker-containers-on-the-new-controller-nodes\">Deploy Required Services Docker Containers on the New Controller Nodes<\/a><\/li><li><a href=\"#verify-new-node-addition-to-open-stack\">Verify New Node Addition to OpenStack<\/a><\/li><li><a href=\"#testing-virtual-ip-vip-high-availability-on-controller-nodes\">Testing Virtual IP (VIP) high availability on Controller Nodes<\/a><\/li><li><a href=\"#define-preferred-master-controller-node\">Define Preferred Master Controller Node<\/a><\/li><\/ul><\/li><\/ul><\/nav><\/div>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"use-kolla-ansible-to-add-controller-nodes-into-existing-open-stack-cluster\">Use Kolla-Ansible to Add Controller Nodes into Existing OpenStack Cluster<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"what-is-a-controller-node-and-why-is-it-important\">What is a Controller Node and Why is it Important?<\/h3>\n\n\n\n<p>In an OpenStack environment, a controller node plays a crucial role in managing and coordinating various services that make up the cloud infrastructure. It acts as the central hub, overseeing functions like authentication, API requests, and overall orchestration of resources.<\/p>\n\n\n\n<p>Here&#8217;s why a controller node is important:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Centralized Management<\/strong>: The controller node serves as a centralized point for managing core OpenStack services like Nova (compute), Neutron (networking), Keystone (identity), Glance (image), and others. This centralization simplifies administration and control.<\/li>\n\n\n\n<li><strong>Orchestration<\/strong>: It coordinates actions among different services to ensure they work together seamlessly. For example, when you launch a new virtual machine, the controller node manages the process, instructing the compute nodes to carry out the request.<\/li>\n\n\n\n<li><strong>High Availability<\/strong>: Deploying multiple controller nodes allows for high availability configurations. If one controller node fails, others can take over, ensuring that essential services remain operational, minimizing downtime.<\/li>\n\n\n\n<li><strong>Scalability<\/strong>: As your OpenStack cloud grows, adding more controller nodes becomes important for distributing the management load. This scalability ensures efficient handling of increased workloads and resources.<\/li>\n\n\n\n<li><strong>Security and Identity Management<\/strong>: The controller node is responsible for user authentication and authorization through Keystone. This centralized identity management enhances security by ensuring consistent access controls across the OpenStack environment.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"what-is-the-recommended-number-of-controller-nodes-in-openstack\">What is the recommended number of Controller Nodes in Openstack?<\/h3>\n\n\n\n<p>The recommended number of controller nodes for an OpenStack deployment depends on various factors, including the size of the cloud environment, the anticipated workload, and the desired level of high availability and redundancy. While there isn&#8217;t a one-size-fits-all answer, here are some few key notes to take:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Minimum Configuration<\/strong>: In a small or development environment, a single controller node might be sufficient for running all the required OpenStack services. This, however, lacks redundancy and high availability.<\/li>\n\n\n\n<li><strong>High Availability (HA) Configuration<\/strong>: For production environments where high availability is crucial, a minimum of three controller nodes is often recommended. This allows for redundancy, ensuring that if one controller node fails, the others can continue to operate. This is typically achieved through the use of HAProxy.<\/li>\n\n\n\n<li><strong>Scalability<\/strong>: As your OpenStack environment grows, you can scale the number of controller nodes to handle increased load and provide better performance. Adding more controller nodes allows for a distributed and load-balanced architecture.<\/li>\n\n\n\n<li><strong>Separation of Services<\/strong>: In larger deployments, it&#8217;s common to separate services onto different controller nodes based on their functions. For example, Keystone (identity service) and Horizon (dashboard) might be on one controller node, while Nova (compute) and Neutron (networking) are on another. This separation can help manage resources more efficiently.<\/li>\n\n\n\n<li><strong>Resource Considerations<\/strong>: Ensure that each controller node has sufficient resources (CPU, RAM, and disk space) to handle the services running on it. The specific resource requirements depend on the size of your deployment and the services you&#8217;re running.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"ideal-timing-for-adding-controller-nodes-in-a-production-environment\">Ideal Timing for Adding Controller Nodes in a Production Environment:<\/h3>\n\n\n\n<p>So, what is the ideal time to expand your OpenStack infrastructure?<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Low Traffic Window:<\/strong>\n<ul class=\"wp-block-list\">\n<li>Plan the addition of new controller nodes during a low-traffic window or a maintenance window to minimize the impact on users.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Off-Peak Hours:<\/strong>\n<ul class=\"wp-block-list\">\n<li>If possible, schedule the addition during off-peak hours when user activity is minimal. This helps reduce the impact on ongoing operations.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Backup and Snapshot:<\/strong>\n<ul class=\"wp-block-list\">\n<li>Before making any changes, take a backup or snapshot of critical components, including databases and configurations. This ensures a quick recovery in case of unexpected issues.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Communication:<\/strong>\n<ul class=\"wp-block-list\">\n<li>Notify users and stakeholders in advance about the planned maintenance window, highlighting potential disruptions and assuring them of the temporary nature of the changes.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Monitoring and Testing:<\/strong>\n<ul class=\"wp-block-list\">\n<li>Implement robust monitoring tools to keep a close eye on the existing infrastructure during the addition of new controller nodes. Test the changes in a staging environment before applying them in production.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Rolling Upgrade:<\/strong>\n<ul class=\"wp-block-list\">\n<li>If your OpenStack deployment method supports it, consider a rolling upgrade approach. This involves adding new nodes, ensuring they work seamlessly, and then gradually migrating services to the new nodes without disrupting the entire environment.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Verify High Availability:<\/strong>\n<ul class=\"wp-block-list\">\n<li>If your OpenStack deployment is designed for high availability, make sure that the addition of new controller nodes aligns with the HA configuration. Verify that services can failover and operate as expected.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Rollback Plan:<\/strong>\n<ul class=\"wp-block-list\">\n<li>Have a well-defined rollback plan in case issues arise during the addition of controller nodes. This includes reverting to the previous state and ensuring minimal impact on users.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Post-Deployment Verification:<\/strong>\n<ul class=\"wp-block-list\">\n<li>After the new controller nodes are added, perform thorough testing to ensure that OpenStack services are functioning correctly. Monitor the environment to identify and address any issues promptly.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Documentation:<\/strong>\n<ul class=\"wp-block-list\">\n<li>Update documentation to reflect the changes made, including details about the new controller nodes. This helps maintain a clear record for future reference.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<p>So, how can you add a controller node(s) into an existing OpenStack using Kolla-Ansible;<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"prepare-the-nodes-for-addition-into-open-stack\">Prepare the Nodes for Addition into OpenStack<\/h3>\n\n\n\n<p>When using Kolla-Ansible for deployment, most of the pre-requisites will be taken care by Kolla-Ansible.<\/p>\n\n\n\n<p>You however need to do the fresh installation of the OS, initial IP assignent to the node, hostname, creation of first user accounts&#8230; (You can automate if you want).<\/p>\n\n\n\n<p>Also, I would recommend that you use same OS version for uniformity across the cluster and easy management. We are running Ubuntu 22.04 LTS<\/p>\n\n\n\n<p>Based on our basic deployment architecture;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>------------------+---------------------------------------------+--------------------------------+----------------------------------+\n                  |                                             |                                |                                  |\n+-----------------+-------------------------+     +-------------+-------------+     +------------+--------------+     +-------------+-------------+\n|        [ Controller Node ]                |     |     [ Compute01 Node ]    |     |    [ Storage01 Node ]     |     |     [ Compute02 Node ]    |\n|                                           |     |                           |     |                           |     |                           |\n|        br0: VIP and Mgt IP                |     |  enp1s0: 192.168.200.202  |     |  enp1s0: 192.168.200.201  |     |  enp1s0: 192.168.200.203  |\n|             VIP: 192.168.200.254          |     |  enp2s0: 10.100.0.110\/24  |     |                           |     |  enp2s0: 10.100.0.111\/24  |\n|             Mgt IP: 192.168.200.200       |     +---------------------------+     +---------------------------+     +---------------------------+\n|        br-ex: Provider Network            |\n|               10.100.0.100\/24             |\n+-------------------------------------------+\n<\/code><\/pre>\n\n\n\n<p>In this guide, we have added two controller nodes, assigned the IP addresses, create required user account with required sudo rights. Our basic architecture will now look like;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>------------------+---------------------------------------------+--------------------------------+----------------------------------+\n                  |                                             |                                |                                  |\n+-----------------+-------------------------+     +-------------+-------------+     +------------+--------------+     +-------------+-------------+\n|        [  Controller 01  ]                |     |     [ Compute01 Node ]    |     |    [ Storage01 Node ]     |     |     [ Compute02 Node ]    |\n|                                           |     |                           |     |                           |     |                           |\n|        br0: VIP and Mgt IP                |     |  enp1s0: 192.168.200.202  |     |  enp1s0: 192.168.200.201  |     |  enp1s0: 192.168.200.203  |\n|             VIP: 192.168.200.254          |     |  enp2s0: 10.100.0.110\/24  |     |                           |     |  enp2s0: 10.100.0.111\/24  |\n|             Mgt IP: 192.168.200.200       |     +---------------------------+     +---------------------------+     +---------------------------+\n|        br-ex: Provider Network            |\n|               10.100.0.100\/24             |\n+-----------------+-------------------------+\n                  |\n+-----------------+-------------------------+\n|        [  Controller 02  ]                |\n|                                           |\n|        br0: Mgt IP: 192.168.200.204       |\n|        br-ex: Provider Network            |\n|               10.100.0.101\/24             |\n+-----------------+-------------------------+\n                  |\n+-----------------+-------------------------+\n|        [  Controller 03  ]                |\n|                                           |\n|        br0: Mgt IP: 192.168.200.205       |\n|        br-ex: Provider Network            |\n|               10.100.0.102\/24             |\n+-----------------+-------------------------+\n<\/code><\/pre>\n\n\n\n<p>Check our <a href=\"https:\/\/kifarunix.com\/deploy-multinode-openstack-using-kolla-ansible\/#network-configurations-on-the-nodes\" target=\"_blank\" rel=\"noreferrer noopener\">controller nodes network configuration<\/a>.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"are-you-using-shared-storage-for-glance-images\">Are you using Shared Storage for Glance Images?<\/h3>\n\n\n\n<p>Well, chances are you are using a shared storage for storing glance images. If by any chance this is the case in your environment, then you need to ensure that all controller nodes have access to the shared storage.<\/p>\n\n\n\n<p>For example, in our demo environment, we are using NFS share for storing Glance images;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>(kolla-ansible) kifarunix@controller01:~$ <strong>docker inspect glance_api<\/strong><\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>           {\n                \"Type\": \"bind\",\n<strong>                \"Source\": \"\/mnt\/glance\",\n                \"Destination\": \"\/var\/lib\/glance\",<\/strong>\n                \"Mode\": \"rw\",\n                \"RW\": true,\n                \"Propagation\": \"rprivate\"\n            }\n        ],\n<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>df -hT -P \/mnt\/glance<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>Filesystem                  Type  Size  Used Avail Use% Mounted on\n192.168.200.201:\/mnt\/glance <strong>nfs4<\/strong>  100G  747M  100G   1% \/mnt\/glance<\/code><\/pre>\n\n\n\n<p>Hence, configure the new controller nodes to also access the share by updating their FSTAB files accordingly.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo vim \/etc\/fstab<\/code><\/pre>\n\n\n\n<p>Add the line to mount the appropriate NFS share;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>192.168.200.201:\/mnt\/glance \/mnt\/glance nfs _netdev,defaults 0 0<\/code><\/pre>\n\n\n\n<p>Where:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><code>192.168.200.201:\/mnt\/glance<\/code>: This part indicates the NFS server&#8217;s IP address (192.168.200.201) and the exported path (\/mnt\/glance) on the NFS server.<\/li>\n\n\n\n<li><code>\/mnt\/glance<\/code>: This is the local mount point on the current machine where the NFS share will be mounted.<\/li>\n\n\n\n<li><code>nfs<\/code>: This specifies the file system type to be mounted, in this case, NFS (Network File System).<\/li>\n\n\n\n<li><code>_netdev<\/code>: This option indicates that the filesystem is a network device and should be mounted after the network has been enabled. This is typically used for network file systems like NFS.<\/li>\n\n\n\n<li><code>defaults<\/code>: This includes a set of default mount options. The specific default options depend on the operating system, but they typically include options for read\/write access, user\/group permissions, etc.<\/li>\n\n\n\n<li><code>0<\/code>: The dump parameter. It is used by the <code>dump<\/code> utility to determine whether the file system needs to be backed up.<\/li>\n\n\n\n<li><code>0<\/code>: The pass parameter. It is used by the <code>fsck<\/code> utility to determine the order in which file systems should be checked.<\/li>\n<\/ul>\n\n\n\n<p>Save and exit the file.<\/p>\n\n\n\n<p>Ensure that the NFS client packages are installed on your system.<\/p>\n\n\n\n<p>On Debian\/Ubuntu, you might need to install the <code>nfs-common<\/code> package:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo apt update\nsudo apt install nfs-common<\/code><\/pre>\n\n\n\n<p>On Red Hat-based systems, you might need to install the <code>nfs-utils<\/code> package:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo yum install nfs-utils<\/code><\/pre>\n\n\n\n<p>Then mount the share;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code> sudo mount -a<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"copy-deployment-user-ssh-keys-from-control-node-to-new-controller-node\">Copy Deployment User SSH Keys from Control Node to New Controller Node<\/h3>\n\n\n\n<p>Your control node is the node where you are running Kolla-ansible. In our setup, we are running Kolla-ansible in our controller01 node.<\/p>\n\n\n\n<p>In regards to the deployment user, if you are not using SSH keys, you need to define the username and password in the multinode configuration file for the new respective controller node to define how Kolla-Ansible will login to configure that respective node.<\/p>\n\n\n\n<p>We are using SSH keys on our guide which we already generated while <a href=\"https:\/\/kifarunix.com\/deploy-multinode-openstack-using-kolla-ansible\/#create-kolla-ansible-deployment-user-account\" target=\"_blank\" rel=\"noreferrer noopener\">creating Kolla-Ansible Deployment User Account<\/a>.<\/p>\n\n\n\n<p>Hence, let just copy the SSH keys into the new controller nodes.<\/p>\n\n\n\n<p>First of all, let&#8217;s ensure the new controller nodes are reachable via their hostnames from the control node.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo tee -a \/etc\/hosts &lt;&lt; EOL\n192.168.200.204 controller02\n192.168.200.205 controller03\nEOL<\/code><\/pre>\n\n\n\n<p>Let&#8217;s confirm reachability;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ping controller02 -c 4<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>PING controller02 (192.168.200.204) 56(84) bytes of data.\n64 bytes from controller02 (192.168.200.204): icmp_seq=1 ttl=64 time=0.260 ms\n64 bytes from controller02 (192.168.200.204): icmp_seq=2 ttl=64 time=0.286 ms\n64 bytes from controller02 (192.168.200.204): icmp_seq=3 ttl=64 time=0.334 ms\n64 bytes from controller02 (192.168.200.204): icmp_seq=4 ttl=64 time=0.312 ms\n\n--- controller02 ping statistics ---\n4 packets transmitted, 4 received, 0% packet loss, time 3056ms\nrtt min\/avg\/max\/mdev = 0.260\/0.298\/0.334\/0.027 ms\n<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>ping controller03 -c 4<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>PING controller03 (192.168.200.205) 56(84) bytes of data.\n64 bytes from controller03 (192.168.200.205): icmp_seq=1 ttl=64 time=0.452 ms\n64 bytes from controller03 (192.168.200.205): icmp_seq=2 ttl=64 time=0.397 ms\n64 bytes from controller03 (192.168.200.205): icmp_seq=3 ttl=64 time=0.398 ms\n64 bytes from controller03 (192.168.200.205): icmp_seq=4 ttl=64 time=0.487 ms\n\n--- controller03 ping statistics ---\n4 packets transmitted, 4 received, 0% packet loss, time 3076ms\nrtt min\/avg\/max\/mdev = 0.397\/0.433\/0.487\/0.038 ms\n<\/code><\/pre>\n\n\n\n<p>Next, copy the keys;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>for i in 02 03; do ssh-copy-id kifarunix@controller$i; done<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"update-kolla-ansible-inventory\">Update Kolla-Ansible Inventory<\/h3>\n\n\n\n<p>Next, you need to add the new controller nodes in the inventory file. Since we are running a multinode deployment, open the <strong>multinode<\/strong> inventory and add your new controller node.<\/p>\n\n\n\n<p>This is a snippet of how our multinode inventory looks like before we add the new controller node;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>cat multinode<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code># These initial groups are the only groups required to be modified. The\n# additional groups are for more control of the environment.\n[control]\ncontroller01 ansible_connection=local neutron_external_interface=vethext\n\n# The above can also be specified as follows:\n#control[01:03]     ansible_user=kolla\n\n# The network nodes are where your l3-agent and loadbalancers will run\n# This can be the same as a host in the control group\n[network]\ncontroller01 ansible_connection=local neutron_external_interface=vethext network_interface=br0\n\n[compute]\ncompute01 neutron_external_interface=enp2s0 network_interface=enp1s0\ncompute02 neutron_external_interface=enp2s0 network_interface=enp1s0\n\n\n[monitoring]\ncontroller01 ansible_connection=local neutron_external_interface=vethext\n\n# When compute nodes and control nodes use different interfaces,\n# you need to comment out \"api_interface\" and other interfaces from the globals.yml\n# and specify like below:\n#compute01 neutron_external_interface=eth0 api_interface=em1 tunnel_interface=em1\n\n[storage]\nstorage01 neutron_external_interface=enp10s0 network_interface=enp1s0\n\n[deployment]\nlocalhost       ansible_connection=local\n\n[baremetal:children]\ncontrol\nnetwork\ncompute\nstorage\nmonitoring\n\n[tls-backend:children]\ncontrol\n\n# You can explicitly specify which hosts run each project by updating the\n# groups in the sections below. Common services are grouped together.\n\n[common:children]\ncontrol\nnetwork\ncompute\nstorage\nmonitoring\n\n[collectd:children]\ncompute\n\n[grafana:children]\nmonitoring\n\n[etcd:children]\ncontrol\n\n[influxdb:children]\nmonitoring\n\n[prometheus:children]\nmonitoring\n\n[kafka:children]\ncontrol\n\n[telegraf:children]\ncompute\ncontrol\nmonitoring\nnetwork\nstorage\n\n[hacluster:children]\ncontrol\n\n[hacluster-remote:children]\ncompute\n\n[loadbalancer:children]\nnetwork\n\n[mariadb:children]\ncontrol\n\n[rabbitmq:children]\ncontrol\n\n[outward-rabbitmq:children]\ncontrol\n\n[monasca-agent:children]\ncompute\ncontrol\nmonitoring\nnetwork\nstorage\n\n[monasca:children]\nmonitoring\n\n[storm:children]\nmonitoring\n\n[keystone:children]\ncontrol\n\n[glance:children]\ncontrol\n\n[nova:children]\ncontrol\n\n[neutron:children]\nnetwork\n\n[openvswitch:children]\nnetwork\ncompute\nmanila-share\n\n[cinder:children]\ncontrol\n\n[cloudkitty:children]\ncontrol\n\n[freezer:children]\ncontrol\n\n[memcached:children]\ncontrol\n\n[horizon:children]\ncontrol\n\n[swift:children]\ncontrol\n\n[barbican:children]\ncontrol\n\n[heat:children]\ncontrol\n\n[murano:children]\ncontrol\n\n[solum:children]\ncontrol\n\n[ironic:children]\ncontrol\n\n[magnum:children]\ncontrol\n\n[sahara:children]\ncontrol\n\n[mistral:children]\ncontrol\n\n[manila:children]\ncontrol\n\n[ceilometer:children]\ncontrol\n\n[aodh:children]\ncontrol\n\n[cyborg:children]\ncontrol\ncompute\n\n[gnocchi:children]\ncontrol\n\n[tacker:children]\ncontrol\n\n[trove:children]\ncontrol\n\n[senlin:children]\ncontrol\n\n[vitrage:children]\ncontrol\n\n[watcher:children]\ncontrol\n\n[octavia:children]\ncontrol\n\n[designate:children]\ncontrol\n\n[placement:children]\ncontrol\n\n[bifrost:children]\ndeployment\n\n[zookeeper:children]\ncontrol\n\n[zun:children]\ncontrol\n\n[skyline:children]\ncontrol\n\n[redis:children]\ncontrol\n\n[blazar:children]\ncontrol\n\n[venus:children]\nmonitoring\n\n# Additional control implemented here. These groups allow you to control which\n# services run on which hosts at a per-service level.\n#\n# Word of caution: Some services are required to run on the same host to\n# function appropriately. For example, neutron-metadata-agent must run on the\n# same host as the l3-agent and (depending on configuration) the dhcp-agent.\n\n# Common\n[cron:children]\ncommon\n\n[fluentd:children]\ncommon\n\n[kolla-logs:children]\ncommon\n\n[kolla-toolbox:children]\ncommon\n\n[opensearch:children]\ncontrol\n\n# Opensearch dashboards\n[opensearch-dashboards:children]\nopensearch\n\n# Glance\n[glance-api:children]\nglance\n\n# Nova\n[nova-api:children]\nnova\n\n[nova-conductor:children]\nnova\n\n[nova-super-conductor:children]\nnova\n\n[nova-novncproxy:children]\nnova\n\n[nova-scheduler:children]\nnova\n\n[nova-spicehtml5proxy:children]\nnova\n\n[nova-compute-ironic:children]\nnova\n\n[nova-serialproxy:children]\nnova\n\n# Neutron\n[neutron-server:children]\ncontrol\n\n[neutron-dhcp-agent:children]\nneutron\n\n[neutron-l3-agent:children]\nneutron\n\n[neutron-metadata-agent:children]\nneutron\n\n[neutron-ovn-metadata-agent:children]\ncompute\nnetwork\n\n[neutron-bgp-dragent:children]\nneutron\n\n[neutron-infoblox-ipam-agent:children]\nneutron\n\n[neutron-metering-agent:children]\nneutron\n\n[ironic-neutron-agent:children]\nneutron\n\n[neutron-ovn-agent:children]\ncompute\nnetwork\n\n# Cinder\n[cinder-api:children]\ncinder\n\n[cinder-backup:children]\nstorage\n\n[cinder-scheduler:children]\ncinder\n\n[cinder-volume:children]\nstorage\n\n# Cloudkitty\n[cloudkitty-api:children]\ncloudkitty\n\n[cloudkitty-processor:children]\ncloudkitty\n\n# Freezer\n[freezer-api:children]\nfreezer\n\n[freezer-scheduler:children]\nfreezer\n\n# iSCSI\n[iscsid:children]\ncompute\nstorage\nironic\n\n[tgtd:children]\nstorage\n\n# Manila\n[manila-api:children]\nmanila\n\n[manila-scheduler:children]\nmanila\n\n[manila-share:children]\nnetwork\n\n[manila-data:children]\nmanila\n\n# Swift\n[swift-proxy-server:children]\nswift\n\n[swift-account-server:children]\nstorage\n\n[swift-container-server:children]\nstorage\n\n[swift-object-server:children]\nstorage\n\n# Barbican\n[barbican-api:children]\nbarbican\n\n[barbican-keystone-listener:children]\nbarbican\n\n[barbican-worker:children]\nbarbican\n\n# Heat\n[heat-api:children]\nheat\n\n[heat-api-cfn:children]\nheat\n\n[heat-engine:children]\nheat\n\n# Murano\n[murano-api:children]\nmurano\n\n[murano-engine:children]\nmurano\n\n# Monasca\n[monasca-agent-collector:children]\nmonasca-agent\n\n[monasca-agent-forwarder:children]\nmonasca-agent\n\n[monasca-agent-statsd:children]\nmonasca-agent\n\n[monasca-api:children]\nmonasca\n\n[monasca-log-persister:children]\nmonasca\n\n[monasca-log-metrics:children]\nmonasca\n\n[monasca-thresh:children]\nmonasca\n\n[monasca-notification:children]\nmonasca\n\n[monasca-persister:children]\nmonasca\n\n# Storm\n[storm-worker:children]\nstorm\n\n[storm-nimbus:children]\nstorm\n\n# Ironic\n[ironic-api:children]\nironic\n\n[ironic-conductor:children]\nironic\n\n[ironic-inspector:children]\nironic\n\n[ironic-tftp:children]\nironic\n\n[ironic-http:children]\nironic\n\n# Magnum\n[magnum-api:children]\nmagnum\n\n[magnum-conductor:children]\nmagnum\n\n# Sahara\n[sahara-api:children]\nsahara\n\n[sahara-engine:children]\nsahara\n\n# Solum\n[solum-api:children]\nsolum\n\n[solum-worker:children]\nsolum\n\n[solum-deployer:children]\nsolum\n\n[solum-conductor:children]\nsolum\n\n[solum-application-deployment:children]\nsolum\n\n[solum-image-builder:children]\nsolum\n\n# Mistral\n[mistral-api:children]\nmistral\n\n[mistral-executor:children]\nmistral\n\n[mistral-engine:children]\nmistral\n\n[mistral-event-engine:children]\nmistral\n\n# Ceilometer\n[ceilometer-central:children]\nceilometer\n\n[ceilometer-notification:children]\nceilometer\n\n[ceilometer-compute:children]\ncompute\n\n[ceilometer-ipmi:children]\ncompute\n\n# Aodh\n[aodh-api:children]\naodh\n\n[aodh-evaluator:children]\naodh\n\n[aodh-listener:children]\naodh\n\n[aodh-notifier:children]\naodh\n\n# Cyborg\n[cyborg-api:children]\ncyborg\n\n[cyborg-agent:children]\ncompute\n\n[cyborg-conductor:children]\ncyborg\n\n# Gnocchi\n[gnocchi-api:children]\ngnocchi\n\n[gnocchi-statsd:children]\ngnocchi\n\n[gnocchi-metricd:children]\ngnocchi\n\n# Trove\n[trove-api:children]\ntrove\n\n[trove-conductor:children]\ntrove\n\n[trove-taskmanager:children]\ntrove\n\n# Multipathd\n[multipathd:children]\ncompute\nstorage\n\n# Watcher\n[watcher-api:children]\nwatcher\n\n[watcher-engine:children]\nwatcher\n\n[watcher-applier:children]\nwatcher\n\n# Senlin\n[senlin-api:children]\nsenlin\n\n[senlin-conductor:children]\nsenlin\n\n[senlin-engine:children]\nsenlin\n\n[senlin-health-manager:children]\nsenlin\n\n# Octavia\n[octavia-api:children]\noctavia\n\n[octavia-driver-agent:children]\noctavia\n\n[octavia-health-manager:children]\noctavia\n\n[octavia-housekeeping:children]\noctavia\n\n[octavia-worker:children]\noctavia\n\n# Designate\n[designate-api:children]\ndesignate\n\n[designate-central:children]\ndesignate\n\n[designate-producer:children]\ndesignate\n\n[designate-mdns:children]\nnetwork\n\n[designate-worker:children]\ndesignate\n\n[designate-sink:children]\ndesignate\n\n[designate-backend-bind9:children]\ndesignate\n\n# Placement\n[placement-api:children]\nplacement\n\n# Zun\n[zun-api:children]\nzun\n\n[zun-wsproxy:children]\nzun\n\n[zun-compute:children]\ncompute\n\n[zun-cni-daemon:children]\ncompute\n\n# Skyline\n[skyline-apiserver:children]\nskyline\n\n[skyline-console:children]\nskyline\n\n# Tacker\n[tacker-server:children]\ntacker\n\n[tacker-conductor:children]\ntacker\n\n# Vitrage\n[vitrage-api:children]\nvitrage\n\n[vitrage-notifier:children]\nvitrage\n\n[vitrage-graph:children]\nvitrage\n\n[vitrage-ml:children]\nvitrage\n\n[vitrage-persistor:children]\nvitrage\n\n# Blazar\n[blazar-api:children]\nblazar\n\n[blazar-manager:children]\nblazar\n\n# Prometheus\n[prometheus-node-exporter:children]\nmonitoring\ncontrol\ncompute\nnetwork\nstorage\n\n[prometheus-mysqld-exporter:children]\nmariadb\n\n[prometheus-haproxy-exporter:children]\nloadbalancer\n\n[prometheus-memcached-exporter:children]\nmemcached\n\n[prometheus-cadvisor:children]\nmonitoring\ncontrol\ncompute\nnetwork\nstorage\n\n[prometheus-alertmanager:children]\nmonitoring\n\n[prometheus-openstack-exporter:children]\nmonitoring\n\n[prometheus-elasticsearch-exporter:children]\nopensearch\n\n[prometheus-blackbox-exporter:children]\nmonitoring\n\n[prometheus-libvirt-exporter:children]\ncompute\n\n[prometheus-msteams:children]\nprometheus-alertmanager\n\n[masakari-api:children]\ncontrol\n\n[masakari-engine:children]\ncontrol\n\n[masakari-hostmonitor:children]\ncontrol\n\n[masakari-instancemonitor:children]\ncompute\n\n[ovn-controller:children]\novn-controller-compute\novn-controller-network\n\n[ovn-controller-compute:children]\ncompute\n\n[ovn-controller-network:children]\nnetwork\n\n[ovn-database:children]\ncontrol\n\n[ovn-northd:children]\novn-database\n\n[ovn-nb-db:children]\novn-database\n\n[ovn-sb-db:children]\novn-database\n\n[venus-api:children]\nvenus\n\n[venus-manager:children]\nvenus\n<\/code><\/pre>\n\n\n\n<p>So, we will update the <strong>[control]<\/strong> group to add our new nodes such that the configuration looks like;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>vim multinode<\/code><\/pre>\n\n\n\n<p>See <strong><code>controller[02:03]<\/code><\/strong>.<\/p>\n\n\n\n<pre class=\"scroll-box\"><code># These initial groups are the only groups required to be modified. The\n# additional groups are for more control of the environment.\n[control]\ncontroller01 ansible_connection=local neutron_external_interface=vethext\n<strong>control[02:03] neutron_external_interface=vethext<\/strong>\n\n# The network nodes are where your l3-agent and loadbalancers will run\n# This can be the same as a host in the control group\n[network]\ncontroller01 ansible_connection=local neutron_external_interface=vethext network_interface=br0\n<strong>controller[02:03] neutron_external_interface=vethext network_interface=br0<\/strong>\n\n[compute]\ncompute01 neutron_external_interface=enp2s0 network_interface=enp1s0\ncompute02 neutron_external_interface=enp2s0 network_interface=enp1s0\n\n[monitoring]\ncontroller01 ansible_connection=local neutron_external_interface=vethext\n<strong>controller[02:03] neutron_external_interface=vethext<\/strong>\n\n# When controller nodes and control nodes use different interfaces,\n# you need to comment out \"api_interface\" and other interfaces from the globals.yml\n# and specify like below:\n#compute01 neutron_external_interface=eth0 api_interface=em1 tunnel_interface=em1\n\n[storage]\nstorage01 neutron_external_interface=enp10s0 network_interface=enp1s0\n\n[deployment]\nlocalhost       ansible_connection=local\n\n[baremetal:children]\ncontrol\nnetwork\ncompute\nstorage\nmonitoring\n\n[tls-backend:children]\ncontrol\n\n# You can explicitly specify which hosts run each project by updating the\n# groups in the sections below. Common services are grouped together.\n\n[common:children]\ncontrol\nnetwork\ncompute\nstorage\nmonitoring\n...\n<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"enable-ha-proxy-for-high-availability-and-load-balancing\">Enable HAProxy for High Availability and Load Balancing<\/h3>\n\n\n\n<p>The <code>globals.yaml<\/code> file in Kolla-Ansible is a configuration file where you can set various global parameters for your OpenStack deployment. The parameter <code>enable_haproxy<\/code> in <code>globals.yaml<\/code> is used to specify whether or not HAProxy should be enabled as part of the deployment.<\/p>\n\n\n\n<p>Set the value of <strong>enable_haproxy<\/strong> to <strong>yes<\/strong> to configure HAProxy to provide load balancing and high availability for your OpenStack services across the controller nodes.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>vim \/etc\/kolla\/globals.yml<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>...\n<strong>enable_haproxy: \"yes\"<\/strong>\nenable_keepalived: \"{{ enable_haproxy | bool }}\"\n...<\/code><\/pre>\n\n\n\n<p>Save and exit the file.<\/p>\n\n\n\n<p>The above configuration basically sets the value of <code>enable_keepalived<\/code> to a boolean indicating that if HAProxy is enabled (<code>enable_haproxy<\/code> is &#8220;yes&#8221;), <code>enable_keepalived<\/code> will be <code>True<\/code>; otherwise, it will be <code>False<\/code>.<\/p>\n\n\n\n<p>While Keepalived manages virtual IP addresses associated with the active controller node using VRRP (Virtual Router Redundancy Protocol), HAProxy load-balances traffic to service backends.<\/p>\n\n\n\n<p>HAProxy regularly checks the health of each controller node by sending health-check requests. If a node becomes unreachable or fails these health checks, HAProxy stops directing traffic to that node.<\/p>\n\n\n\n<p>Keepalived continuously monitors the health of the active controller node. If it detects a failure (for example, if the controller node becomes unreachable), Keepalived triggers a failover process and automatically transfers the VIP to one of the standby controller nodes that are still healthy based on their priority setting.<\/p>\n\n\n\n<p>You can check the configuration of HAProxy on <code><strong>\/etc\/kolla\/haproxy\/haproxy.cfg<\/strong><\/code> and configuration for KeepAlived on <strong><code>\/etc\/kolla\/keepalived\/keepalived.conf<\/code><\/strong>.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"activate-kolla-ansible-virtual-environment\">Activate Kolla-Ansible Virtual Environment<\/h3>\n\n\n\n<p>Activate your respective virtual environment;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>source ~\/kolla-ansible\/bin\/activate<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"test-connectivity-to-the-node\">Test Connectivity to the Node<\/h3>\n\n\n\n<p>Execute the Ansible command below to check the reachability of node in your inventory using the Ansible <code>ping<\/code> module.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ansible -i multinode -m ping controller02,controller03<\/code><\/pre>\n\n\n\n<p>Sample output;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>controller03 | SUCCESS => {\n    \"ansible_facts\": {\n        \"discovered_interpreter_python\": \"\/usr\/bin\/python3\"\n    },\n    \"changed\": false,\n    \"ping\": \"pong\"\n}\ncontroller02 | SUCCESS => {\n    \"ansible_facts\": {\n        \"discovered_interpreter_python\": \"\/usr\/bin\/python3\"\n    },\n    \"changed\": false,\n    \"ping\": \"pong\"\n}\n<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"bootstrap-the-new-controller-nodes\">Bootstrap the new Controller Nodes<\/h3>\n\n\n\n<p>You need to bootstrap the controller nodes with kolla deploy dependencies by running the command below.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kolla-ansible -i &lt;inventory&gt; bootstrap-servers &#91; --limit &lt;limit&gt; ]<\/code><\/pre>\n\n\n\n<p>Replace the <strong><code>&lt;inventory&gt;<\/code><\/strong> with your inventory file. When adding controller nodes, <strong><code>&lt;limit&gt;<\/code><\/strong> needs to specify the controller nodes group, in this case <strong>control<\/strong>, since you need the boostrap command to generate the Fernet keys that are used for encrypting tokens in Keystone and provide security for authentication tokens and then distributed them across the Keystone hosts\/controller nodes to ensure consistency.<\/p>\n\n\n\n<p>Be cautious about re-bootstrapping a cloud that has already been boostrapped. See some <a href=\"https:\/\/docs.openstack.org\/kolla-ansible\/latest\/reference\/deployment-and-bootstrapping\/bootstrap-servers.html#rebootstrapping\" target=\"_blank\" rel=\"noreferrer noopener\">considerations for reboostrapping<\/a>.<\/p>\n\n\n\n<p>Thus, our command will look like;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kolla-ansible -i multinode bootstrap-servers --limit control<\/code><\/pre>\n\n\n\n<p>This command may restart docker containers in other nodes. Hence, also check the ideal time to add controller nodes into production deployment as outlined above.<\/p>\n\n\n\n<p>If at some point during the deployment, MariaDB stucks at starting after a restart;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>docker ps | grep mariadb<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>fa28f34cbad9   quay.io\/openstack.kolla\/mariadb-server:2023.1-ubuntu-jammy                  \"dumb-init -- kolla_\u2026\"   3 hours ago     Up 24 seconds (health: starting)             mariadb<\/code><\/pre>\n\n\n\n<p>And such errors appear on the logs;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>tail -f \/var\/log\/kolla\/mariadb\/mariadb.log<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>2023-11-12  8:19:18 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT3.50534S), skipping check\n2023-11-12  8:19:47 0 [Note] WSREP: PC protocol downgrade 1 -> 0\n2023-11-12  8:19:47 0 [Note] WSREP: view((empty))\n2023-11-12  8:19:47 0 [ERROR] WSREP: failed to open gcomm backend connection: 110: failed to reach primary view: 110 (Connection timed out)\n\t at .\/gcomm\/src\/pc.cpp:connect():160\n2023-11-12  8:19:47 0 [ERROR] WSREP: .\/gcs\/src\/gcs_core.cpp:gcs_core_open():221: Failed to open backend connection: -110 (Connection timed out)\n2023-11-12  8:19:48 0 [ERROR] WSREP: .\/gcs\/src\/gcs.cpp:gcs_open():1669: Failed to open channel 'openstack' at 'gcomm:\/\/192.168.200.200:4567,192.168.200.204:4567,192.168.200.205:4567': -110 (Connection timed out)\n2023-11-12  8:19:48 0 [ERROR] WSREP: gcs connect failed: Connection timed out\n2023-11-12  8:19:48 0 [ERROR] WSREP: wsrep::connect(gcomm:\/\/192.168.200.200:4567,192.168.200.204:4567,192.168.200.205:4567) failed: 7\n2023-11-12  8:19:48 0 [ERROR] Aborting\n231112 08:19:48 mysqld_safe mysqld from pid file \/var\/lib\/mysql\/mariadb.pid ended\n<\/code><\/pre>\n\n\n\n<p>You can execure the database recovery;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kolla-ansible -i multinode mariadb_recovery<\/code><\/pre>\n\n\n\n<p>Once that is completed, you can re-run the deployment command.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"run-pre-deployment-checks-on-the-new-controller-nodes\">Run Pre-Deployment Checks on the New Controller Nodes<\/h3>\n\n\n\n<p>Next, run pre-deployment checks for node;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kolla-ansible -i multinode prechecks --limit controller02,controller03<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"deploy-required-services-docker-containers-on-the-new-controller-nodes\">Deploy Required Services Docker Containers on the New Controller Nodes<\/h3>\n\n\n\n<p>Next, deploy Docker containers for the required services on the new controller nodes.<\/p>\n\n\n\n<p>To begin with, download the container images into the host<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kolla-ansible -i multinode pull --limit controller02,controller03<\/code><\/pre>\n\n\n\n<p>When the command completes, you can list the container images on the node.<\/p>\n\n\n\n<p>You can login to the node and check list images;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>docker images<\/code><\/pre>\n\n\n\n<p>or just list them using ansible from the control node;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ansible -i multinode -m raw -a \"sudo docker images\" controller02<\/code><\/pre>\n\n\n\n<p>Sample output;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>controller02 | CHANGED | rc=0 >>\nREPOSITORY                                              TAG                   IMAGE ID       CREATED       SIZE\nquay.io\/openstack.kolla\/cinder-scheduler                2023.1-ubuntu-jammy   41d2c120dcaf   5 hours ago   1.33GB\nquay.io\/openstack.kolla\/cinder-api                      2023.1-ubuntu-jammy   dc89b0b74a64   5 hours ago   1.33GB\nquay.io\/openstack.kolla\/neutron-server                  2023.1-ubuntu-jammy   dd92163f698e   5 hours ago   1.05GB\nquay.io\/openstack.kolla\/neutron-l3-agent                2023.1-ubuntu-jammy   47e8b6290070   5 hours ago   1.05GB\nquay.io\/openstack.kolla\/aodh-listener                   2023.1-ubuntu-jammy   4f103cd2aa96   5 hours ago   892MB\nquay.io\/openstack.kolla\/aodh-notifier                   2023.1-ubuntu-jammy   577c5461b901   5 hours ago   892MB\nquay.io\/openstack.kolla\/aodh-api                        2023.1-ubuntu-jammy   80311624cca1   5 hours ago   892MB\nquay.io\/openstack.kolla\/aodh-evaluator                  2023.1-ubuntu-jammy   82850f3f9b82   5 hours ago   892MB\nquay.io\/openstack.kolla\/neutron-dhcp-agent              2023.1-ubuntu-jammy   3c9ab4533eb5   5 hours ago   1.04GB\nquay.io\/openstack.kolla\/keystone                        2023.1-ubuntu-jammy   d4c445793a75   5 hours ago   942MB\nquay.io\/openstack.kolla\/keystone-ssh                    2023.1-ubuntu-jammy   946e8484c8a5   5 hours ago   948MB\nquay.io\/openstack.kolla\/neutron-openvswitch-agent       2023.1-ubuntu-jammy   8c8468ded47c   5 hours ago   1.04GB\nquay.io\/openstack.kolla\/neutron-metadata-agent          2023.1-ubuntu-jammy   87d457f01029   5 hours ago   1.04GB\nquay.io\/openstack.kolla\/keystone-fernet                 2023.1-ubuntu-jammy   0d92e47e97b5   5 hours ago   946MB\nquay.io\/openstack.kolla\/nova-novncproxy                 2023.1-ubuntu-jammy   14934aa7795f   5 hours ago   1.22GB\nquay.io\/openstack.kolla\/placement-api                   2023.1-ubuntu-jammy   fb19639b8875   5 hours ago   894MB\nquay.io\/openstack.kolla\/horizon                         2023.1-ubuntu-jammy   2c893a67ae9f   5 hours ago   1.11GB\nquay.io\/openstack.kolla\/heat-api-cfn                    2023.1-ubuntu-jammy   7ec11e8fefb9   5 hours ago   960MB\nquay.io\/openstack.kolla\/heat-api                        2023.1-ubuntu-jammy   e925a6c8b000   5 hours ago   960MB\nquay.io\/openstack.kolla\/heat-engine                     2023.1-ubuntu-jammy   048d6ebe056e   5 hours ago   960MB\nquay.io\/openstack.kolla\/nova-scheduler                  2023.1-ubuntu-jammy   ab1b447c7021   5 hours ago   1.11GB\nquay.io\/openstack.kolla\/nova-conductor                  2023.1-ubuntu-jammy   ca809956e88b   5 hours ago   1.11GB\nquay.io\/openstack.kolla\/nova-api                        2023.1-ubuntu-jammy   137618290a77   5 hours ago   1.11GB\nquay.io\/openstack.kolla\/glance-api                      2023.1-ubuntu-jammy   4e47347129e3   5 hours ago   1.04GB\nquay.io\/openstack.kolla\/zun-wsproxy                     2023.1-ubuntu-jammy   6004d566ab20   5 hours ago   1.01GB\nquay.io\/openstack.kolla\/zun-api                         2023.1-ubuntu-jammy   f93535599eba   5 hours ago   1.01GB\nquay.io\/openstack.kolla\/gnocchi-api                     2023.1-ubuntu-jammy   4e8e97ea301a   5 hours ago   1.07GB\nquay.io\/openstack.kolla\/gnocchi-metricd                 2023.1-ubuntu-jammy   c8cefb46f796   5 hours ago   1.07GB\nquay.io\/openstack.kolla\/gnocchi-statsd                  2023.1-ubuntu-jammy   b6b3c5404e55   5 hours ago   1.07GB\nquay.io\/openstack.kolla\/ceilometer-central              2023.1-ubuntu-jammy   52e724bda206   5 hours ago   895MB\nquay.io\/openstack.kolla\/ceilometer-notification         2023.1-ubuntu-jammy   24affe5317f4   5 hours ago   895MB\nquay.io\/openstack.kolla\/kolla-toolbox                   2023.1-ubuntu-jammy   47fb2da898a6   5 hours ago   819MB\nquay.io\/openstack.kolla\/mariadb-server                  2023.1-ubuntu-jammy   08f31f386d7b   5 hours ago   605MB\nquay.io\/openstack.kolla\/mariadb-clustercheck            2023.1-ubuntu-jammy   df9693cf6795   5 hours ago   322MB\nquay.io\/openstack.kolla\/prometheus-blackbox-exporter    2023.1-ubuntu-jammy   3786777ba92f   5 hours ago   278MB\nquay.io\/openstack.kolla\/prometheus-alertmanager         2023.1-ubuntu-jammy   9dc45a653636   5 hours ago   314MB\nquay.io\/openstack.kolla\/prometheus-node-exporter        2023.1-ubuntu-jammy   29854ef51590   5 hours ago   276MB\nquay.io\/openstack.kolla\/prometheus-memcached-exporter   2023.1-ubuntu-jammy   bccdd901d45b   5 hours ago   271MB\nquay.io\/openstack.kolla\/prometheus-mysqld-exporter      2023.1-ubuntu-jammy   142d34487ae1   5 hours ago   272MB\nquay.io\/openstack.kolla\/prometheus-openstack-exporter   2023.1-ubuntu-jammy   00708fff486b   5 hours ago   267MB\nquay.io\/openstack.kolla\/prometheus-cadvisor             2023.1-ubuntu-jammy   57e64a35a36d   5 hours ago   295MB\nquay.io\/openstack.kolla\/prometheus-haproxy-exporter     2023.1-ubuntu-jammy   8da898f5509f   5 hours ago   272MB\nquay.io\/openstack.kolla\/prometheus-v2-server            2023.1-ubuntu-jammy   34d5a456575e   5 hours ago   469MB\nquay.io\/openstack.kolla\/openvswitch-vswitchd            2023.1-ubuntu-jammy   da575b11b8d8   5 hours ago   274MB\nquay.io\/openstack.kolla\/grafana                         2023.1-ubuntu-jammy   9485136dab77   5 hours ago   673MB\nquay.io\/openstack.kolla\/openvswitch-db-server           2023.1-ubuntu-jammy   45b86c2748cb   5 hours ago   274MB\nquay.io\/openstack.kolla\/fluentd                         2023.1-ubuntu-jammy   f0b67ea51417   5 hours ago   529MB\nquay.io\/openstack.kolla\/keepalived                      2023.1-ubuntu-jammy   34c6f0412dca   5 hours ago   269MB\nquay.io\/openstack.kolla\/rabbitmq                        2023.1-ubuntu-jammy   a9e21182aae7   5 hours ago   314MB\nquay.io\/openstack.kolla\/cron                            2023.1-ubuntu-jammy   f0bc61f82d64   5 hours ago   258MB\nquay.io\/openstack.kolla\/memcached                       2023.1-ubuntu-jammy   366a6292a66e   5 hours ago   259MB\nquay.io\/openstack.kolla\/etcd                            2023.1-ubuntu-jammy   37bc7fc289fe   5 hours ago   297MB\nquay.io\/openstack.kolla\/haproxy                         2023.1-ubuntu-jammy   b64892cee984   5 hours ago   266MB\n<\/code><\/pre>\n\n\n\n<p>Deploy containers on the new controller nodes. Again, you have to specify all the controller nodes here. Thus, <strong>&#8211;limit control<\/strong>.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kolla-ansible -i multinode deploy --limit control<\/code><\/pre>\n\n\n\n<p>Ensure there is no error. If any, fix it and proceed.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"verify-new-node-addition-to-open-stack\">Verify New Node Addition to OpenStack<\/h3>\n\n\n\n<p>You can now verify if the new controller nodes have been successfully added into OpenStack cluster.<\/p>\n\n\n\n<p>To begin with, you can list Docker containers running on the node (the command below is executed from the control node);<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ansible -i multinode -m raw -a \"sudo docker ps\" controller02<\/code><\/pre>\n\n\n\n<p>sample output;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>controller02 | CHANGED | rc=0 >>\nCONTAINER ID   IMAGE                                                                       COMMAND                  CREATED       STATUS                   PORTS     NAMES\nae81cdd54526   quay.io\/openstack.kolla\/zun-wsproxy:2023.1-ubuntu-jammy                     \"dumb-init --single-\u2026\"   2 hours ago   Up 2 hours (healthy)               zun_wsproxy\n559251e9067c   quay.io\/openstack.kolla\/zun-api:2023.1-ubuntu-jammy                         \"dumb-init --single-\u2026\"   2 hours ago   Up 2 hours (healthy)               zun_api\n703c89f0d130   quay.io\/openstack.kolla\/grafana:2023.1-ubuntu-jammy                         \"dumb-init --single-\u2026\"   2 hours ago   Up 2 hours                         grafana\nd55aa3751b1d   quay.io\/openstack.kolla\/aodh-notifier:2023.1-ubuntu-jammy                   \"dumb-init --single-\u2026\"   2 hours ago   Up 2 hours (healthy)               aodh_notifier\nb7f1ade8cf50   quay.io\/openstack.kolla\/aodh-listener:2023.1-ubuntu-jammy                   \"dumb-init --single-\u2026\"   2 hours ago   Up 2 hours (healthy)               aodh_listener\n5a2f03854727   quay.io\/openstack.kolla\/aodh-evaluator:2023.1-ubuntu-jammy                  \"dumb-init --single-\u2026\"   2 hours ago   Up 2 hours (healthy)               aodh_evaluator\nca5054ffcb1c   quay.io\/openstack.kolla\/aodh-api:2023.1-ubuntu-jammy                        \"dumb-init --single-\u2026\"   2 hours ago   Up 2 hours (healthy)               aodh_api\nb3fe5de9c0d7   quay.io\/openstack.kolla\/ceilometer-central:2023.1-ubuntu-jammy              \"dumb-init --single-\u2026\"   2 hours ago   Up 2 hours (unhealthy)             ceilometer_central\n96e41a027685   quay.io\/openstack.kolla\/ceilometer-notification:2023.1-ubuntu-jammy         \"dumb-init --single-\u2026\"   2 hours ago   Up 2 hours (healthy)               ceilometer_notification\n407a3b083aa9   quay.io\/openstack.kolla\/gnocchi-statsd:2023.1-ubuntu-jammy                  \"dumb-init --single-\u2026\"   2 hours ago   Up 2 hours (healthy)               gnocchi_statsd\n8847afd527c6   quay.io\/openstack.kolla\/gnocchi-metricd:2023.1-ubuntu-jammy                 \"dumb-init --single-\u2026\"   2 hours ago   Up 2 hours (healthy)               gnocchi_metricd\n6409e89ac7c9   quay.io\/openstack.kolla\/gnocchi-api:2023.1-ubuntu-jammy                     \"dumb-init --single-\u2026\"   2 hours ago   Up 2 hours (healthy)               gnocchi_api\n9f89e8ea58cd   quay.io\/openstack.kolla\/horizon:2023.1-ubuntu-jammy                         \"dumb-init --single-\u2026\"   2 hours ago   Up 2 hours (healthy)               horizon\n853c7bbaea53   quay.io\/openstack.kolla\/mariadb-server:2023.1-ubuntu-jammy                  \"dumb-init -- kolla_\u2026\"   2 hours ago   Up 2 hours (healthy)               mariadb\n76651604bdf0   quay.io\/openstack.kolla\/heat-engine:2023.1-ubuntu-jammy                     \"dumb-init --single-\u2026\"   7 hours ago   Up 2 hours (healthy)               heat_engine\nc6ef2875a2b4   quay.io\/openstack.kolla\/heat-api-cfn:2023.1-ubuntu-jammy                    \"dumb-init --single-\u2026\"   7 hours ago   Up 2 hours (healthy)               heat_api_cfn\n12dd70306101   quay.io\/openstack.kolla\/heat-api:2023.1-ubuntu-jammy                        \"dumb-init --single-\u2026\"   7 hours ago   Up 2 hours (healthy)               heat_api\n2cddac565e6d   quay.io\/openstack.kolla\/neutron-metadata-agent:2023.1-ubuntu-jammy          \"dumb-init --single-\u2026\"   7 hours ago   Up 2 hours (healthy)               neutron_metadata_agent\nae50f1adecbe   quay.io\/openstack.kolla\/neutron-l3-agent:2023.1-ubuntu-jammy                \"dumb-init --single-\u2026\"   7 hours ago   Up 2 hours (healthy)               neutron_l3_agent\na14026c69184   quay.io\/openstack.kolla\/neutron-dhcp-agent:2023.1-ubuntu-jammy              \"dumb-init --single-\u2026\"   7 hours ago   Up 2 hours (healthy)               neutron_dhcp_agent\n803595b10a5f   quay.io\/openstack.kolla\/neutron-openvswitch-agent:2023.1-ubuntu-jammy       \"dumb-init --single-\u2026\"   7 hours ago   Up 2 hours (healthy)               neutron_openvswitch_agent\nbc1c0f6b268d   quay.io\/openstack.kolla\/neutron-server:2023.1-ubuntu-jammy                  \"dumb-init --single-\u2026\"   7 hours ago   Up 2 hours (healthy)               neutron_server\nc0cd45da353d   quay.io\/openstack.kolla\/openvswitch-vswitchd:2023.1-ubuntu-jammy            \"dumb-init --single-\u2026\"   7 hours ago   Up 2 hours (healthy)               openvswitch_vswitchd\n4d452111b3ba   quay.io\/openstack.kolla\/openvswitch-db-server:2023.1-ubuntu-jammy           \"dumb-init --single-\u2026\"   7 hours ago   Up 2 hours (healthy)               openvswitch_db\n33cb79ec0a55   quay.io\/openstack.kolla\/nova-novncproxy:2023.1-ubuntu-jammy                 \"dumb-init --single-\u2026\"   7 hours ago   Up 2 hours (healthy)               nova_novncproxy\n6ff84d56ce4c   quay.io\/openstack.kolla\/nova-conductor:2023.1-ubuntu-jammy                  \"dumb-init --single-\u2026\"   7 hours ago   Up 2 hours (healthy)               nova_conductor\na6d168bca4c9   quay.io\/openstack.kolla\/nova-api:2023.1-ubuntu-jammy                        \"dumb-init --single-\u2026\"   7 hours ago   Up 2 hours (healthy)               nova_api\ne338faeeeee0   quay.io\/openstack.kolla\/nova-scheduler:2023.1-ubuntu-jammy                  \"dumb-init --single-\u2026\"   7 hours ago   Up 2 hours (healthy)               nova_scheduler\n9988ebb9618a   quay.io\/openstack.kolla\/placement-api:2023.1-ubuntu-jammy                   \"dumb-init --single-\u2026\"   7 hours ago   Up 2 hours (healthy)               placement_api\n9a6e5585150f   quay.io\/openstack.kolla\/cinder-scheduler:2023.1-ubuntu-jammy                \"dumb-init --single-\u2026\"   7 hours ago   Up 2 hours (healthy)               cinder_scheduler\n77d6e4115252   quay.io\/openstack.kolla\/cinder-api:2023.1-ubuntu-jammy                      \"dumb-init --single-\u2026\"   7 hours ago   Up 2 hours (healthy)               cinder_api\ne0590534d6c8   quay.io\/openstack.kolla\/glance-api:2023.1-ubuntu-jammy                      \"dumb-init --single-\u2026\"   7 hours ago   Up 2 hours (healthy)               glance_api\ne08bb362be08   quay.io\/openstack.kolla\/keystone:2023.1-ubuntu-jammy                        \"dumb-init --single-\u2026\"   7 hours ago   Up 2 hours (healthy)               keystone\n21534f07df6a   quay.io\/openstack.kolla\/keystone-fernet:2023.1-ubuntu-jammy                 \"dumb-init --single-\u2026\"   7 hours ago   Up 2 hours (healthy)               keystone_fernet\n06726634c6eb   quay.io\/openstack.kolla\/keystone-ssh:2023.1-ubuntu-jammy                    \"dumb-init --single-\u2026\"   7 hours ago   Up 2 hours (healthy)               keystone_ssh\n8fba633812b7   quay.io\/openstack.kolla\/etcd:2023.1-ubuntu-jammy                            \"dumb-init --single-\u2026\"   7 hours ago   Up 2 hours                         etcd\nbce5a05dcbba   quay.io\/openstack.kolla\/rabbitmq:2023.1-ubuntu-jammy                        \"dumb-init --single-\u2026\"   7 hours ago   Up 2 hours (healthy)               rabbitmq\nf1ff17d3b41a   quay.io\/openstack.kolla\/prometheus-blackbox-exporter:2023.1-ubuntu-jammy    \"dumb-init --single-\u2026\"   7 hours ago   Up 2 hours                         prometheus_blackbox_exporter\n5d900639936d   quay.io\/openstack.kolla\/prometheus-openstack-exporter:2023.1-ubuntu-jammy   \"dumb-init --single-\u2026\"   7 hours ago   Up 2 hours                         prometheus_openstack_exporter\n0f5bd3e873b0   quay.io\/openstack.kolla\/prometheus-alertmanager:2023.1-ubuntu-jammy         \"dumb-init --single-\u2026\"   7 hours ago   Up 2 hours                         prometheus_alertmanager\naa9ef3408ee0   quay.io\/openstack.kolla\/prometheus-cadvisor:2023.1-ubuntu-jammy             \"dumb-init --single-\u2026\"   7 hours ago   Up 2 hours                         prometheus_cadvisor\n85443d572966   quay.io\/openstack.kolla\/prometheus-memcached-exporter:2023.1-ubuntu-jammy   \"dumb-init --single-\u2026\"   7 hours ago   Up 2 hours                         prometheus_memcached_exporter\n77df867d2b80   quay.io\/openstack.kolla\/prometheus-haproxy-exporter:2023.1-ubuntu-jammy     \"dumb-init --single-\u2026\"   7 hours ago   Up 2 hours                         prometheus_haproxy_exporter\nf4e0e360bdad   quay.io\/openstack.kolla\/prometheus-mysqld-exporter:2023.1-ubuntu-jammy      \"dumb-init --single-\u2026\"   7 hours ago   Up 2 hours                         prometheus_mysqld_exporter\n1394ba79ee7c   quay.io\/openstack.kolla\/prometheus-node-exporter:2023.1-ubuntu-jammy        \"dumb-init --single-\u2026\"   7 hours ago   Up 2 hours                         prometheus_node_exporter\n43136ca82078   quay.io\/openstack.kolla\/prometheus-v2-server:2023.1-ubuntu-jammy            \"dumb-init --single-\u2026\"   7 hours ago   Up 2 hours                         prometheus_server\n87595629490c   quay.io\/openstack.kolla\/memcached:2023.1-ubuntu-jammy                       \"dumb-init --single-\u2026\"   7 hours ago   Up 2 hours (healthy)               memcached\n184454c7d93b   quay.io\/openstack.kolla\/mariadb-clustercheck:2023.1-ubuntu-jammy            \"dumb-init --single-\u2026\"   7 hours ago   Up 2 hours                         mariadb_clustercheck\n27690e469dc2   quay.io\/openstack.kolla\/keepalived:2023.1-ubuntu-jammy                      \"dumb-init --single-\u2026\"   7 hours ago   Up 2 hours                         keepalived\n0d61e2f1edc6   quay.io\/openstack.kolla\/haproxy:2023.1-ubuntu-jammy                         \"dumb-init --single-\u2026\"   7 hours ago   Up 2 hours (healthy)               haproxy\nb9477be14773   quay.io\/openstack.kolla\/cron:2023.1-ubuntu-jammy                            \"dumb-init --single-\u2026\"   7 hours ago   Up 2 hours                         cron\n291f972ce16c   quay.io\/openstack.kolla\/kolla-toolbox:2023.1-ubuntu-jammy                   \"dumb-init --single-\u2026\"   7 hours ago   Up 2 hours                         kolla_toolbox\n9e3bffb135fa   quay.io\/openstack.kolla\/fluentd:2023.1-ubuntu-jammy                         \"dumb-init --single-\u2026\"   7 hours ago   Up 2 hours                         fluentd\n<\/code><\/pre>\n\n\n\n<p>Well, I am not sure there is an easy way to list the controller nodes, -:). But from what I can see, they are listed under internal availability zone.<\/p>\n\n\n\n<p>Load the credentials;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>source $HOME\/kolla-ansible\/bin\/activate<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>source \/etc\/kolla\/admin-openrc.sh<\/code><\/pre>\n\n\n\n<p>Let&#8217;s see;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>openstack availability zone list --compute --long<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>+-----------+-------------+---------------+--------------+----------------+----------------------------------------+\n| Zone Name | Zone Status | Zone Resource | Host Name    | Service Name   | Service Status                         |\n+-----------+-------------+---------------+--------------+----------------+----------------------------------------+\n| internal  | available   |               | controller02 | nova-scheduler | enabled :-) 2023-11-14T15:22:10.000000 |\n| internal  | available   |               | controller02 | nova-conductor | enabled :-) 2023-11-14T15:22:14.000000 |\n| internal  | available   |               | controller03 | nova-scheduler | enabled :-) 2023-11-14T15:22:11.000000 |\n| internal  | available   |               | controller03 | nova-conductor | enabled :-) 2023-11-14T15:22:11.000000 |\n| internal  | available   |               | controller01 | nova-scheduler | enabled :-) 2023-11-14T15:22:13.000000 |\n| internal  | available   |               | controller01 | nova-conductor | enabled :-) 2023-11-14T15:22:12.000000 |\n| nova      | available   |               | compute02    | nova-compute   | enabled :-) 2023-11-14T15:22:09.000000 |\n| nova      | available   |               | compute01    | nova-compute   | enabled :-) 2023-11-14T15:22:07.000000 |\n+-----------+-------------+---------------+--------------+----------------+----------------------------------------+\n<\/code><\/pre>\n\n\n\n<p>If you can also list compute services, you will see some services distributed across controller nodes!<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>openstack compute service list<\/code><\/pre>\n\n\n\n<p>Output;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>+--------------------------------------+----------------+--------------+----------+---------+-------+----------------------------+\n| ID                                   | Binary         | Host         | Zone     | Status  | State | Updated At                 |\n+--------------------------------------+----------------+--------------+----------+---------+-------+----------------------------+\n| b4d36484-cd27-4f5b-bf18-c93d7184890d | nova-scheduler | controller01 | internal | enabled | up    | 2023-11-12T15:42:45.000000 |\n| 7be8d02c-8a76-48a0-bfa0-3edfc583bc9c | nova-scheduler | controller02 | internal | enabled | up    | 2023-11-12T15:42:44.000000 |\n| 24253619-07d6-479f-975c-b2876c81d12f | nova-scheduler | controller03 | internal | enabled | up    | 2023-11-12T15:42:42.000000 |\n| 5efddcac-fdf4-4ce3-8843-43e1784dc8d2 | nova-conductor | controller01 | internal | enabled | up    | 2023-11-12T15:42:48.000000 |\n| 77ea6f87-5144-476b-9685-ebb6f1765b09 | nova-compute   | compute02    | nova     | enabled | up    | 2023-11-12T15:42:42.000000 |\n| 546d891d-04f8-41f7-b2a6-714569e4bc52 | nova-compute   | compute01    | nova     | enabled | up    | 2023-11-12T15:42:46.000000 |\n| 3d1fa2a8-0637-4c39-9941-d80a011a6be1 | nova-conductor | controller03 | internal | enabled | up    | 2023-11-12T15:42:42.000000 |\n| 22df4a17-2d4b-4413-b81e-32856c5ba2c5 | nova-conductor | controller02 | internal | enabled | up    | 2023-11-12T15:42:42.000000 |\n+--------------------------------------+----------------+--------------+----------+---------+-------+----------------------------+\n<\/code><\/pre>\n\n\n\n<p>You can see some services are distributed across the controller nodes.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"testing-virtual-ip-vip-high-availability-on-controller-nodes\">Testing Virtual IP (VIP) high availability on Controller Nodes<\/h3>\n\n\n\n<p>At this point, you might need to simulate a scenario whereby the active node (containing the VIP) fails, and verifying that the failover process works as expected, with minimal or no disruption to the services.<\/p>\n\n\n\n<p>Each of our three controller nodes, have their priority numbers defined.<\/p>\n\n\n\n<p>What is a priority number in Keepalived? In Keepalived, the <strong><code>priority<\/code><\/strong> number is used to determine the priority of a node in a High Availability (HA) setup. The node with the highest priority is typically chosen as the master (active) node, and in the event of a failure, the node with the next highest priority becomes the master. The priority is an integer value within the range of 0 to 255, and the node with the highest priority is considered the most preferred for the master role.<\/p>\n\n\n\n<p>Let&#8217;s check our nodes priorities;<\/p>\n\n\n\n<p>Controller01;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo grep priority \/etc\/kolla\/keepalived\/keepalived.conf<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>    priority 1<\/code><\/pre>\n\n\n\n<p>Controller02;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo grep priority \/etc\/kolla\/keepalived\/keepalived.conf<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>    priority 2<\/code><\/pre>\n\n\n\n<p>Controller03;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo grep priority \/etc\/kolla\/keepalived\/keepalived.conf<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>    priority 3<\/code><\/pre>\n\n\n\n<p>From the commands output above, controller03 is the most preferred node and for our case, it currently has the VIP assigned;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>root@controller03:~# ip -br a\nlo               UNKNOWN        127.0.0.1\/8 ::1\/128 \nenp1s0           UP             \nenp7s0           UP             \nbr-ex            UP             10.100.0.102\/24 fe80::c866:d5ff:feba:3cf8\/64 \nbr0              UP             192.168.200.205\/24 <strong>192.168.200.254\/32<\/strong> fe80::f04e:aaff:fe5c:90f9\/64 \nvethext@vethint  UP             fe80::44ea:9aff:fe7f:b1e0\/64 \nvethint@vethext  UP             \novs-system       DOWN           \nbr-int           DOWN           \nbr-tun           DOWN           \nroot@controller03:~#\n<\/code><\/pre>\n\n\n\n<p>So, there are multiple ways to simulate the failover here. For example, you can pause node from the virtualization host, you can take down the network interface, or anything to make sure it temporarily becomes unreachable. If you are doing this in production environment, be cautious!!<\/p>\n\n\n\n<p>So, before I take down controller03, I will tail the logs on controller01 and controller02;<\/p>\n\n\n\n<p>Controller02;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>root@controller02:~# <strong>docker logs --tail 10 -f keepalived<\/strong><\/code><\/pre>\n\n\n\n<p>Controller01;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kifarunix@controller01:~$ <strong>docker logs --tail 10 -f keepalived<\/strong><\/code><\/pre>\n\n\n\n<p>Now, temporarily taking down controller03;<\/p>\n\n\n\n<p>Logs on Controller02 (see the last line, <strong><em>Sun Nov 12 16:46:39 2023: (kolla_internal_vip_51) Entering MASTER STATE<\/em><\/strong>) with higher priority than controller01;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>Sun Nov 12 12:58:27 2023:    Reset ARP config counter 0\nSun Nov 12 12:58:27 2023:    Original arp_ignore 0\nSun Nov 12 12:58:27 2023:    Original arp_filter 0\nSun Nov 12 12:58:27 2023:    Original promote_secondaries 1\nSun Nov 12 12:58:27 2023:    Reset promote_secondaries counter 0\nSun Nov 12 12:58:27 2023: Script `check_alive` now returning 1\nSun Nov 12 12:58:27 2023: VRRP_Script(check_alive) failed (exited with status 1)\nSun Nov 12 12:58:31 2023: Script `check_alive` now returning 0\nSun Nov 12 12:58:49 2023: VRRP_Script(check_alive) succeeded\nSun Nov 12 12:58:49 2023: (kolla_internal_vip_51) Entering BACKUP STATE\n\n\n\nSun Nov 12 16:46:39 2023: (kolla_internal_vip_51) Entering MASTER STATE\n<\/code><\/pre>\n\n\n\n<p>Nothing much on controller01.<\/p>\n\n\n\n<p>Hence, controller02 should now be having the VIP;<\/p>\n\n\n\n<pre class=\"scroll-box\"><code>root@controller02:~# ip -br a\nlo               UNKNOWN        127.0.0.1\/8 ::1\/128 \nenp1s0           UP             \nenp7s0           UP             \nbr-ex            UP             10.100.0.101\/24 fe80::5cfc:faff:fe5e:9a93\/64 \nbr0              UP             192.168.200.204\/24 <strong>192.168.200.254\/32<\/strong> fe80::b04a:bbff:fece:1c6\/64 \nvethext@vethint  UP             fe80::d458:43ff:fe9c:23a2\/64 \nvethint@vethext  UP             \novs-system       DOWN           \nbr-int           DOWN           \nbr-tun           DOWN           \nvxlan_sys_4789   UNKNOWN        fe80::98c7:18ff:fe88:4f14\/64 \n<\/code><\/pre>\n\n\n\n<p>And all services are working as expected for me! Hence, at this point, I believe it is good to conclude that the three node controller cluster is working as expected.<\/p>\n\n\n\n<p>When controller03, with high priority is back up, it will assume the master state.<\/p>\n\n\n\n<p>Controller02 Keepalived logs;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Sun Nov 12 16:56:52 2023: (kolla_internal_vip_51) Master received advert from 192.168.200.205 with higher priority 3, ours 2\nSun Nov 12 16:56:52 2023: (kolla_internal_vip_51) Entering BACKUP STATE<\/code><\/pre>\n\n\n\n<p>Heads up! Before you can conclude that your cluster is working as expected, perform <strong>thorough<\/strong> testing!<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"define-preferred-master-controller-node\">Define Preferred Master Controller Node<\/h3>\n\n\n\n<p>To define a preferred master controller node in a Keepalived setup, you typically need to adjust the priority configuration for the nodes. In Keepalived, the node with the highest priority is elected as the master.<\/p>\n\n\n\n<p>Here is how to define preferred master controller node:<\/p>\n\n\n\n<p>Identify the Keepalived configuration file on each controller node. In our Kolla-ansible deployment, the Keepalived configuration is found under <code><strong>\/etc\/kolla<\/strong><\/code>.<\/p>\n\n\n\n<p><strong><code>\/etc\/kolla\/keepalived\/keepalived.conf<\/code><\/strong> is the configuration.<\/p>\n\n\n\n<p>Edit the Keepalived configuration file on the node you want to set as the preferred master. Look for the <code>vrrp_instance<\/code> section, and specifically, the <code>priority<\/code> parameter.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>vim \/etc\/kolla\/keepalived\/keepalived.conf<\/code><\/pre>\n\n\n\n<p> Change the <code>priority<\/code> value to a higher number for the preferred master node. Nodes with higher priority values will be preferred for the master role. We set the value of controller02 priority to 20. <\/p>\n\n\n\n<pre class=\"scroll-box\"><code>vrrp_script check_alive {\n    script \"\/check_alive.sh\"\n    interval 2\n    fall 2\n    rise 10\n}\n\nvrrp_instance kolla_internal_vip_51 {\n    state BACKUP\n    nopreempt\n    interface br0\n    virtual_router_id 51\n    priority 20\n    advert_int 1\n    virtual_ipaddress {\n        192.168.200.254 dev br0\n    }\n    authentication {\n        auth_type PASS\n        auth_pass 9MC1BOSy764sBcZFHxzniiLwMrBz6iz3HiRcOLWv\n    }\n    track_script {\n        check_alive\n    }\n}\n<\/code><\/pre>\n\n\n\n<p>Save the changes to the configuration file.<\/p>\n\n\n\n<p>Restart Keepalived on the node where you made the configuration changes:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo docker restart keepalived<\/code><\/pre>\n\n\n\n<p>Check the logs or status to confirm that Keepalived has successfully restarted.<\/p>\n\n\n\n<p>On the node that currently has VIP, you can restart Keepalived to relieve it of master role.<\/p>\n\n\n\n<p>Check that the controller node with high priority has assumed the master role:<\/p>\n\n\n\n<p>Sample logs on my controller node;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>docker logs --tail 5 -f keepalived<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>Wed Nov 15 05:19:20 2023:    Original arp_filter 0\nWed Nov 15 05:19:20 2023:    Original promote_secondaries 1\nWed Nov 15 05:19:20 2023:    Reset promote_secondaries counter 0\nWed Nov 15 05:19:20 2023: VRRP_Script(check_alive) succeeded\nWed Nov 15 05:19:20 2023: (kolla_internal_vip_51) Entering BACKUP STATE\n<strong>Wed Nov 15 05:19:39 2023: (kolla_internal_vip_51) Entering MASTER STATE<\/strong>\n<\/code><\/pre>\n\n\n\n<p>Check the Virtual IP (VIP) to ensure it is associated with the preferred master node:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>root@controller02:~# ip a<\/code><\/pre>\n\n\n\n<pre class=\"scroll-box\"><code>...\n2: enp1s0: &lt;BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br0 state UP group default qlen 1000\n    link\/ether 52:54:00:0c:c4:bf brd ff:ff:ff:ff:ff:ff\n3: enp7s0: &lt;BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br-ex state UP group default qlen 1000\n    link\/ether 52:54:00:5c:50:31 brd ff:ff:ff:ff:ff:ff\n4: br-ex: &lt;BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000\n    link\/ether 5e:fc:fa:5e:9a:93 brd ff:ff:ff:ff:ff:ff\n    inet 10.100.0.101\/24 brd 10.100.0.255 scope global br-ex\n       valid_lft forever preferred_lft forever\n    inet6 fe80::5cfc:faff:fe5e:9a93\/64 scope link \n       valid_lft forever preferred_lft forever\n5: br0: &lt;BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000\n    link\/ether b2:4a:bb:ce:01:c6 brd ff:ff:ff:ff:ff:ff\n    inet 192.168.200.204\/24 brd 192.168.200.255 scope global br0\n       valid_lft forever preferred_lft forever\n<strong>    inet 192.168.200.254\/32 scope global br0<\/strong>\n       valid_lft forever preferred_lft forever\n    inet6 fe80::b04a:bbff:fece:1c6\/64 scope link \n       valid_lft forever preferred_lft forever\n...\n<\/code><\/pre>\n\n\n\n<p>Remember to repeat these steps for each controller node where you want to set or adjust the priority. Adjust the <code>priority<\/code> values based on your desired preference.<\/p>\n\n\n\n<p>And that concludes our guide on how to add controller nodes into existing Openstack cluster using Kolla-Ansible.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Is it possible to add controller nodes into existing OpenStack cluster using Kolla-Ansible? Of course, yes! In this blog post, you will learn how to<\/p>\n","protected":false},"author":10,"featured_media":19214,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"rank_math_lock_modified_date":false,"footnotes":""},"categories":[121,1884,992,1885,1886],"tags":[7295,7297,7299,7296,7298],"class_list":["post-19138","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-howtos","category-ansible","category-automation","category-cloud-compute","category-openstack","tag-add-more-controllers-to-openstack","tag-kolla-ansible-openstack","tag-openstack-haproxy","tag-openstack-load-balance-controller-nodes","tag-openstack-vip-keepalived","generate-columns","tablet-grid-50","mobile-grid-100","grid-parent","grid-50","resize-featured-image"],"_links":{"self":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/19138"}],"collection":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/users\/10"}],"replies":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/comments?post=19138"}],"version-history":[{"count":22,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/19138\/revisions"}],"predecessor-version":[{"id":20882,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/19138\/revisions\/20882"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media\/19214"}],"wp:attachment":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media?parent=19138"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/categories?post=19138"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/tags?post=19138"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}