{"id":4022,"date":"2020-06-06T23:07:32","date_gmt":"2020-06-06T20:07:32","guid":{"rendered":"https:\/\/kifarunix.com\/?p=4022"},"modified":"2024-03-14T21:54:11","modified_gmt":"2024-03-14T18:54:11","slug":"configure-highly-available-haproxy-with-keepalived-on-ubuntu","status":"publish","type":"post","link":"https:\/\/kifarunix.com\/configure-highly-available-haproxy-with-keepalived-on-ubuntu\/","title":{"rendered":"Configure Highly Available HAProxy with Keepalived on Ubuntu"},"content":{"rendered":"\n<p>How do I make HAProxy high availability? In this guide, we are going to learn how to configure highly available HAProxy with Keepalived on Ubuntu 20.04. <a href=\"https:\/\/keepalived.readthedocs.io\/en\/latest\/introduction.html\" target=\"_blank\" rel=\"noreferrer noopener\">Keepalived<\/a> can work with HAProxy to provide the failover services to a backup route in the event that a specific route becomes unavailable. This ensures a more robust and scalable high availability environment.<\/p>\n\n\n\n<div class=\"wp-block-rank-math-toc-block\" id=\"rank-math-toc\"><h2>Table of Contents<\/h2><nav><ul><li><a href=\"#configuring-highly-available-ha-proxy-with-keepalived\">Configuring Highly Available HAProxy with Keepalived<\/a><ul><li><a href=\"#install-and-configure-keepalived-on-ubuntu\">Install and Configure Keepalived on Ubuntu<\/a><ul><li><a href=\"#install-keepalived-on-ha-proxy-servers\">Install Keepalived on HAProxy Servers<\/a><\/li><li><a href=\"#configure-ip-forwarding-and-non-local-binding\">Configure IP forwarding and non-local binding<\/a><\/li><li><a href=\"#configure-keepalived\">Configure Keepalived<\/a><\/li><li><a href=\"#running-keepalived-on-ubuntu\">Running Keepalived on Ubuntu<\/a><\/li><\/ul><\/li><li><a href=\"#configure-ha-proxy-with-keepalived-virtual-ip\">Configure HAProxy with Keepalived Virtual IP<\/a><\/li><li><a href=\"#further-reading\">Further Reading<\/a><\/li><li><a href=\"#other-tutorials\">Other tutorials<\/a><\/li><\/ul><\/li><\/ul><\/nav><\/div>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"configuring-highly-available-ha-proxy-with-keepalived\">Configuring Highly Available HAProxy with Keepalived<\/h2>\n\n\n\n<p>While Keepalived uses&nbsp;<code>Linux virtual server<\/code>&nbsp;(LVS) kernel module to perform load balancing and failover tasks on the active and passive routers, HAProxy performs load balancing and high-availability services to TCP and HTTP applications.<\/p>\n\n\n\n<p>Keepalived utilizes Virtual Router Redundancy Protocol to send periodic advertisements between the master (<strong><code>active<\/code><\/strong>) and backup (<code><strong>passive<\/strong><\/code>) LVS routers (which in our case is the HAProxy servers since are load balancing web apps) to determine the state of each other. If a master server fails to advertise itself within a predefined period of time, Keepalived initiates failover and the backup server becomes the master.<\/p>\n\n\n\n<p>All virtual servers are assigned a <code>Virtual IP<\/code>, also known as <code>floating IP<\/code>. This is a publicly routable IP\/address. It is assigned dynamically to an active server at any given time.<\/p>\n\n\n\n<p>This setup requires that you already have an HAProxy server setup and running. We have covered the installation and setup of HAProxy load balancer on various systems in our previous guides;<\/p>\n\n\n\n<p><a rel=\"noreferrer noopener\" href=\"https:\/\/kifarunix.com\/install-and-setup-haproxy-on-ubuntu-20-04\/\" target=\"_blank\">Install and Setup HAProxy on Ubuntu 20.04<\/a><\/p>\n\n\n\n<p><a rel=\"noreferrer noopener\" href=\"https:\/\/kifarunix.com\/install-and-setup-haproxy-on-centos-8\/\" target=\"_blank\">Install and Setup HAProxy on CentOS 8<\/a><\/p>\n\n\n\n<p><a rel=\"noreferrer noopener\" href=\"https:\/\/kifarunix.com\/setup-haproxy-load-balancer-on-fedora-30-fedora-29\/\" target=\"_blank\">Setup HAProxy Load Balancer on Fedora 30\/Fedora 29<\/a><\/p>\n\n\n\n<p><a rel=\"noreferrer noopener\" href=\"https:\/\/kifarunix.com\/configure-haproxy-with-ssl-on-ubuntu-18-04-debian-10-9\/\" target=\"_blank\">Configure HAProxy Load Balancer with SSL on Ubuntu 18.04\/Debian 10\/9<\/a><\/p>\n\n\n\n<p>In this tutorial, we will be using two HAProxy servers with Keepalived for high availability. Below is our deployment architecture.<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"928\" height=\"782\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2020\/06\/haproxy-keepalived.png\" alt=\"Configure Highly Available HAProxy with Keepalived on Ubuntu\" class=\"wp-image-6073\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2020\/06\/haproxy-keepalived.png?v=1591459711 928w, https:\/\/kifarunix.com\/wp-content\/uploads\/2020\/06\/haproxy-keepalived-768x647.png?v=1591459711 768w\" sizes=\"(max-width: 928px) 100vw, 928px\" \/><\/figure><\/div>\n\n\n<p>Assuming that you already have HAProxy setup, proceed with installation and configuration of Keepalived on Ubuntu 20.04.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"install-and-configure-keepalived-on-ubuntu\">Install and Configure Keepalived on Ubuntu<\/h3>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"install-keepalived-on-ha-proxy-servers\"><a href=\"#installkeepaliveonubuntu20.04\">Install Keepalived on HAProxy Servers<\/a><\/h4>\n\n\n\n<p>In our demo environment, we are running HAProxy servers on Ubuntu 20.04. Therefore, assuming your system package cache is up-to-date, run the command below install Keepalived on Ubuntu 20.04<\/p>\n\n\n\n<p>On HAProxy Node 01;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>apt install keepalived<\/code><\/pre>\n\n\n\n<p>Similarly, install Keepalived on second HAProxy server.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>apt install keepalived<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"configure-ip-forwarding-and-non-local-binding\"><a href=\"#ipforwardingandnon-localbind\">Configure IP forwarding and non-local binding<\/a><\/h4>\n\n\n\n<p>To enable Keepalived service to forward network packets to the backend servers, you need to enable IP forwarding. Run this command on both HAProxy servers;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>sed -i 's\/#net.ipv4.ip_forward=1\/net.ipv4.ip_forward=1\/' \/etc\/sysctl.conf<\/code><\/pre>\n\n\n\n<p>Similarly, you need to enable HAProxy and Keepalived to bind to non-local IP address, that is to bind to the failover IP address (Floating IP).<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>echo \"net.ipv4.ip_nonlocal_bind = 1\" &gt;&gt; \/etc\/sysctl.conf<\/code><\/pre>\n\n\n\n<p>Reload sysctl settings;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>sysctl -p<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"configure-keepalived\">Configure Keepalived<\/h4>\n\n\n\n<p>The default configuration file for Keepalived should be <code><strong>\/etc\/keepalived\/keepalived.conf<\/strong><\/code>. However, this configuration is not created by default. Create the configuration with the content below;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code><strong>vim \/etc\/keepalived\/keepalived.conf<\/strong><\/code><\/pre>\n\n\n\n<p><strong>Keepalived Configuration for Master Node (lb01)<\/strong><\/p>\n\n\n\n<pre class=\"scroll-sz\"><code>\n# Global Settings for notifications\nglobal_defs {\n    notification_email {\n        id@domain.com     # Email address for notifications \n    }\n    notification_email_from id@domain.com\t# The from address for the notifications\n    smtp_server 127.0.0.1     \t\t\t# SMTP server address\n    smtp_connect_timeout 15\n}\n  \n# Define the script used to check if haproxy is still working\nvrrp_script chk_haproxy { \n    script \"\/usr\/bin\/killall -0 haproxy\"\n    interval 2 \n    weight 2 \n}\n  \n# Configuration for Virtual Interface\nvrrp_instance LB_VIP {\n    interface enp0s8\n    state MASTER        # set to BACKUP on the peer machine\n    priority 101        # set to  99 on the peer machine\n    virtual_router_id 51\n  \n    smtp_alert          # Enable Notifications Via Email\n  \n    authentication {\n        auth_type AH\n        auth_pass myP@ssword\t# Password for accessing vrrpd. Same on all devices\n    }\n    unicast_src_ip 192.168.57.7 # Private IP address of master\n    unicast_peer {\n        192.168.58.4\t\t# Private IP address of the backup haproxy\n   }\n  \n    # The virtual ip address shared between the two loadbalancers\n    virtual_ipaddress {\n        192.168.100.200\n    }\n     \n    # Use the Defined Script to Check whether to initiate a fail over\n    track_script {\n        chk_haproxy\n    }\n}\n<\/code><\/pre>\n\n\n\n<p><strong>Keepalived Configuration for Backup Node (lb02)<\/strong><\/p>\n\n\n\n<pre class=\"scroll-sz\"><code>\nglobal_defs {\n    notification_email {\n        id@domain.com\n    }\n    notification_email_from id@domain.com\n    smtp_server 127.0.0.1\n    smtp_connect_timeout 15\n}\n  \nvrrp_script chk_haproxy { \n    script \"\/usr\/bin\/killall -0 haproxy\"\n    interval 2 \n    weight 2 \n}\n  \nvrrp_instance LB_VIP {\n    interface enp0s8\n    state BACKUP\n    priority 100\n    virtual_router_id 51\n  \n    smtp_alert\n  \n    authentication {\n        auth_type AH\n        auth_pass myP@ssword\n    }\n    unicast_src_ip 192.168.58.4 # Private IP address of the backup haproxy\n    unicast_peer {\n        192.168.57.7\t# Private IP address of the master haproxy\n   }\n  \n    virtual_ipaddress {\n        192.168.100.200\n    }\n     \n    track_script {\n        chk_haproxy\n    }\n}\n<\/code><\/pre>\n\n\n\n<p>Note that the notification section is optional. You can however follow the link below to install and configure Postfix to use Gmail SMTP as relay;<\/p>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/configure-postfix-to-use-gmail-smtp-on-ubuntu-20-04\/\" target=\"_blank\" rel=\"noreferrer noopener\">Configure Postfix to Use Gmail SMTP on Ubuntu 20.04<\/a><\/p>\n\n\n\n<p>Read more about the configuration parameters used above on <a rel=\"noreferrer noopener\" href=\"https:\/\/www.keepalived.org\/manpage.html\" target=\"_blank\">Keepalived man pages<\/a> and <a rel=\"noreferrer noopener\" href=\"https:\/\/keepalived.readthedocs.io\/en\/latest\/configuration_synopsis.html#\" target=\"_blank\">Keepalived Configuration synopsis<\/a>.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"running-keepalived-on-ubuntu\">Running Keepalived on Ubuntu<\/h4>\n\n\n\n<p>You can now start and enable Keepalived to run on system boot on all nodes;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>systemctl enable --now keepalived<\/code><\/pre>\n\n\n\n<p>Check the status on Master Node;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>systemctl status keepalived<\/code><\/pre>\n\n\n\n<pre class=\"scroll-sz\"><code>\n\u25cf keepalived.service - Keepalive Daemon (LVS and VRRP)\n     Loaded: loaded (\/lib\/systemd\/system\/keepalived.service; enabled; vendor preset: enabled)\n     Active: active (running) since Sat 2020-06-06 18:24:20 UTC; 6s ago\n   Main PID: 7097 (keepalived)\n      Tasks: 2 (limit: 2282)\n     Memory: 2.1M\n     CGroup: \/system.slice\/keepalived.service\n             \u251c\u25007097 \/usr\/sbin\/keepalived --dont-fork\n             \u2514\u25007107 \/usr\/sbin\/keepalived --dont-fork\n\nJun 06 18:24:20 lb01.kifarunix-demo.com Keepalived_vrrp[7107]: (LB_VIP) Initial state master is incompatible with AH authentication - clearing\nJun 06 18:24:20 lb01.kifarunix-demo.com Keepalived_vrrp[7107]: Registering gratuitous ARP shared channel\nJun 06 18:24:20 lb01.kifarunix-demo.com Keepalived_vrrp[7107]: (LB_VIP) Entering BACKUP STATE (init)\nJun 06 18:24:20 lb01.kifarunix-demo.com Keepalived_vrrp[7107]: Remote SMTP server [127.0.0.1]:25 connected.\nJun 06 18:24:20 lb01.kifarunix-demo.com Keepalived_vrrp[7107]: VRRP_Script(chk_haproxy) succeeded\nJun 06 18:24:20 lb01.kifarunix-demo.com Keepalived_vrrp[7107]: (LB_VIP) Changing effective priority from 100 to 102\nJun 06 18:24:20 lb01.kifarunix-demo.com Keepalived_vrrp[7107]: SMTP alert successfully sent.\nJun 06 18:24:23 lb01.kifarunix-demo.com Keepalived_vrrp[7107]: (LB_VIP) Entering MASTER STATE\nJun 06 18:24:23 lb01.kifarunix-demo.com Keepalived_vrrp[7107]: Remote SMTP server [127.0.0.1]:25 connected.\nJun 06 18:24:23 lb01.kifarunix-demo.com Keepalived_vrrp[7107]: SMTP alert successfully sent\n<\/code><\/pre>\n\n\n\n<p>You can as well check the status on the slave node.<\/p>\n\n\n\n<p>Check the IP address assigment;<\/p>\n\n\n\n<p>On the master node;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>ip --brief add<\/code><\/pre>\n\n\n\n<pre class=\"scroll-sz\"><code>\nlo               UNKNOWN        127.0.0.1\/8 ::1\/128 \nenp0s3           UP             10.0.2.15\/24 fe80::a00:27ff:fe9d:888e\/64 \n<strong>enp0s8           UP             192.168.100.81\/24 192.168.100.200\/32 fe80::a00:27ff:feba:9e8c\/64 \n<\/strong>enp0s9           UP             192.168.57.7\/24 fe80::a00:27ff:fe31:b7db\/64\n<\/code><\/pre>\n\n\n\n<p>See the highlighted line, it is assigned a floating IP,  <strong>192.168.100.200\/32<\/strong>.<\/p>\n\n\n\n<p>On the slave;<\/p>\n\n\n\n<pre class=\"scroll-sz\"><code>\nlo               UNKNOWN        127.0.0.1\/8 ::1\/128 \nenp0s3           UP             10.0.2.15\/24 fe80::a00:27ff:fefe:fc06\/64 \nenp0s8           UP             192.168.100.80\/24 fe80::a00:27ff:fe75:9eae\/64 \nenp0s9           UP             192.168.58.4\/24 fe80::a00:27ff:fef4:41\/64\n<\/code><\/pre>\n\n\n\n<p>Now, let us take down interface enp0s8, on the master node. <strong>Be sure to login from the console before you can take the interface down<\/strong>.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>ip link set enp0s8 down<\/code><\/pre>\n\n\n\n<p>Check Keepalived status on the backup node;<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>systemctl status keepalived<\/code><\/pre>\n\n\n\n<pre class=\"scroll-sz\"><code>\n\u25cf keepalived.service - Keepalive Daemon (LVS and VRRP)\n     Loaded: loaded (\/lib\/systemd\/system\/keepalived.service; enabled; vendor preset: enabled)\n     Active: active (running) since Sat 2020-06-06 19:24:27 UTC; 26min ago\n   Main PID: 9948 (keepalived)\n      Tasks: 2 (limit: 2282)\n     Memory: 2.3M\n     CGroup: \/system.slice\/keepalived.service\n             \u251c\u25009948 \/usr\/sbin\/keepalived --dont-fork\n             \u2514\u25009949 \/usr\/sbin\/keepalived --dont-fork\n\nJun 06 19:24:27 lb02.kifarunix-demo.com Keepalived_vrrp[9949]: SECURITY VIOLATION - scripts are being executed but script_security not enabled.\nJun 06 19:24:27 lb02.kifarunix-demo.com Keepalived_vrrp[9949]: Registering gratuitous ARP shared channel\nJun 06 19:24:27 lb02.kifarunix-demo.com Keepalived_vrrp[9949]: (LB_VIP) Entering BACKUP STATE (init)\nJun 06 19:24:27 lb02.kifarunix-demo.com Keepalived_vrrp[9949]: Remote SMTP server [127.0.0.1]:25 connected.\nJun 06 19:24:27 lb02.kifarunix-demo.com Keepalived_vrrp[9949]: VRRP_Script(chk_haproxy) succeeded\nJun 06 19:24:27 lb02.kifarunix-demo.com Keepalived_vrrp[9949]: (LB_VIP) Changing effective priority from 99 to 101\nJun 06 19:24:27 lb02.kifarunix-demo.com Keepalived_vrrp[9949]: SMTP alert successfully sent.\n<strong>Jun 06 19:50:39 lb02.kifarunix-demo.com Keepalived_vrrp[9949]: (LB_VIP) Entering MASTER STATE\n<\/strong>Jun 06 19:50:39 lb02.kifarunix-demo.com Keepalived_vrrp[9949]: Remote SMTP server [127.0.0.1]:25 connected.\nJun 06 19:50:40 lb02.kifarunix-demo.com Keepalived_vrrp[9949]: SMTP alert successfully sent.\n<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"configure-ha-proxy-with-keepalived-virtual-ip\">Configure HAProxy with Keepalived Virtual IP<\/h3>\n\n\n\n<p>The only change that we are going to make on our <a rel=\"noreferrer noopener\" href=\"https:\/\/kifarunix.com\/install-and-setup-haproxy-on-ubuntu-20-04\/#haproxyconfigurationfile\" target=\"_blank\">HAProxy configuration fil<\/a><a href=\"https:\/\/kifarunix.com\/install-and-setup-haproxy-on-ubuntu-20-04\/#haproxyconfigurationfile\" target=\"_blank\" rel=\"noreferrer noopener\">e<\/a> is to adjust the frontend bind IP and the stats IP address to the VIP or the floating IP, which in our case is 192.168.100.200<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>vim \/etc\/haproxy\/haproxy.cfg<\/code><\/pre>\n\n\n\n<pre class=\"scroll-sz\"><code>\n...\nfrontend kifarunixlb\n        bind <strong>192.168.100.200<\/strong>:443 ssl crt \/etc\/ssl\/certs\/haproxy.pem\n        default_backend webapps\n        option forwardfor\n...\nlisten stats\n        bind <strong>192.168.100.200<\/strong>:8443 ssl crt \/etc\/ssl\/certs\/haproxy.pem\n        stats enable\n...\n<\/code><\/pre>\n\n\n\n<p>Save and exit the config file.<\/p>\n\n\n\n<p>Restart HAProxy.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>systemctl restart haproxy<\/code><\/pre>\n\n\n\n<p>You should now be able to access your web apps even when one of the HAProxy servers goes down via the floating IP. Sample screenshot for our setup is in below;<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1292\" height=\"246\" src=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2020\/06\/vip-app01.png\" alt=\"\" class=\"wp-image-6078\" title=\"\" srcset=\"https:\/\/kifarunix.com\/wp-content\/uploads\/2020\/06\/vip-app01.png?v=1591473520 1292w, https:\/\/kifarunix.com\/wp-content\/uploads\/2020\/06\/vip-app01-768x146.png?v=1591473520 768w\" sizes=\"(max-width: 1292px) 100vw, 1292px\" \/><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"further-reading\">Further Reading<\/h3>\n\n\n\n<p><a href=\"https:\/\/www.keepalived.org\/manpage.html\" target=\"_blank\" rel=\"noopener\">Keepalived Documentation<\/a><\/p>\n\n\n\n<p><a rel=\"noreferrer noopener\" href=\"https:\/\/keepalived.readthedocs.io\/en\/latest\/\" target=\"_blank\">Keepalived User Guide<\/a><\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"other-tutorials\">Other tutorials<\/h3>\n\n\n\n<p><a href=\"https:\/\/kifarunix.com\/how-to-install-and-configure-pound-apache-load-balancer-on-ubuntu-16-04\/\" target=\"_blank\" rel=\"noreferrer noopener\">How to Install and Configure Pound as Apache HTTP Load balancer on Ubuntu 16.04<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>How do I make HAProxy high availability? In this guide, we are going to learn how to configure highly available HAProxy with Keepalived on Ubuntu<\/p>\n","protected":false},"author":1,"featured_media":15335,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"rank_math_lock_modified_date":false,"footnotes":""},"categories":[121,1032,92],"tags":[1663,1033,1664,1659,1674,1665,1675,1200],"class_list":["post-4022","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-howtos","category-haproxy","category-load-balancers","tag-configure-haproxy-with-keepalived","tag-haproxy","tag-highly-available-haproxy-with-keepalived","tag-install-haproxy-ubuntu-20-04","tag-install-keepalived-on-ubuntu-20-04","tag-keepalived","tag-setup-haproxy-with-keepalived","tag-ubuntu-20-04","generate-columns","tablet-grid-50","mobile-grid-100","grid-parent","grid-50","resize-featured-image"],"_links":{"self":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/4022"}],"collection":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/comments?post=4022"}],"version-history":[{"count":10,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/4022\/revisions"}],"predecessor-version":[{"id":21430,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/posts\/4022\/revisions\/21430"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media\/15335"}],"wp:attachment":[{"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/media?parent=4022"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/categories?post=4022"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kifarunix.com\/wp-json\/wp\/v2\/tags?post=4022"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}